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1.  This  report  was  prepared  to  provide  general  guidelines  for  a 
system  approach  to  the  design  of  training  programs,  in  order  to  develop 
programs  that  will  be  maximally  effective  and  yet  operate  at  minimum 
cost. 

2.  The  report  is  based  on  an  extensive  literature  survey  and  upon 
training  research  which  has  been  conducted  by  the;  Human  Resources  Research 
Office.  Instructional  system  components  which  are  discussed  include  pre¬ 
sentation  of  information,  practice  techniques,  student  management,  quality 
control,  evaluation  of  the  system,  and  cost  effectiveness. 

3.  This  report,  one  of  a  series  dealing  with  the  technology  of 
training,  will  be  of  interest  to  agencies  or  personnel  concerned  with 
the  design  and  evaluation  of  instructional  systems. 
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FOREWORD 


This  report  presents  a  general  account  of  one  facet  of  the  technology 
for  developing  effective  training,  that  of  designing  instructional  systems. 
The  material  is  based  on  a  survey  of  available  literature,  and  drawn 
particularly  from  Human  Resources  Research  Office  experience  and 
methodology  in  training  research. 

While  it  is  expected  that  many  of  the  ideas  and  techniques  reported 
will  be  superseded  as  research  and  development  for  training  advances, 
the  treatment  of  present  technology  concerning  instructional  system 
design  was  prepared  to  provide  guidelines  to  military  personnel 
currently  involved  with  military  training  programs. 

This  report  is  part  of  a  series  of  closely  related  HumRRO  pub¬ 
lications  by  Robert  G,  Smith,  Jr;  dealing  with  various  aspects  of  the 
technology  of  developing  training.  The  series  includes  The  Develop¬ 
ment  of;  Training  Objectives,  Research  Bulletin  XI,  June  1964;  and 
Controlling  the  Quality  of  Training,  Technical  Report  65-6,  June  1965. 
Each  of  the  technology  reports  is  supplemented  by  an  annotated  bibli¬ 
ography,  which  provides  more  detail  on  the  literature  available  to  the 
instructional  system  designer.  They  are:  An  Annotated  Bibliography 
on  the  Determination  of  Training  Objectives,  Research  Memorandum, 
June  1964;  An  Annotated  Bibliography  on  Proficiency  Measurement 
for  Training  Quality  Control.  Research  Memorandum,  June  1964; 
and  An  Annotated  Bibliography  on  the  Design  of  Instructional  Systems, 
Technical  Report  in  preparation. 

HumRRO  research  efforts  are  conducted  under  Army  Contract 
DA  44-188-ARO-2  and  under  Army  Project  No.  2J024701A712  01, 
Training,  Motivation,  Leadership  Research. 


Meredith  P.  Crawford 
Director 

Human  Resources  Research  Office 


A  training  program  set  up  to  accomplish  a  number  of  specific  objectives  is  composed  of 
many  elements.  These  include  lectures,  demonstrations,  films  and  other  types  of  training  aids 
and  devices,  practical  exercises,  techniques  for  controlling  student  activity  during  training,  and 
tests  for  quality  control.  For  the  training  program  to  be  maximally  effective,  all  these  elements 
must  be  integrated  into  a  system,  designed  to  accomplish  the  objectives  at  minimum  cost.  Various 
principles  and  techniques  of  learning  must  be  applied  in  the  development  of  the  system. 

This  report  is  one  of  the  Human  Resources  Research  Office  publications  on  aspects  of  the 
technology  for  developing  training:  It  summarizes  the  state  of  the  art  at  the  time  of  writing,  and 
is  designed  to  provide  general  guidelines  for  the  design  of  instructional  systems. 


Background 

To  be  certain  the  desired  objectives  are  attained  in  an  efficient  manna,  an  instructional 
system  should  be  built  around  these  essential  functions: . 

(1)  Development  of  training  objectives  based  on  the  job  for  which  the  student 

is  being  trained.  ^ 

(2)  Practice  of  task  performance  until  the  student  has  attained  the  objective. 

(3)  Practice  of  the  knowledges  and  skills  that  are  components  of  the  task. 

(4)  Presentation  of  knowledge  to  the  student. 

(5)  Control  of  student  activity  to  maximally  direct  it  to  learning. 

(6)  Control  of  the  training  quality  by  means  of  a  system  developed  for  this 
specific  purpose. 


Approach 

The  presentation  is  based  on  a  survey  of'  the  available  literature  and  draws  particularly  on 
HumRRO  experience  in  research  studies  on  training.  To  provide  relevant  references  for  a  partic¬ 
ular  topic,  specific  references  ate  cited  far  each  chapter  in  the  Selected  Bibliography  and  are 
supplemented  with  additional  uacitad  references  pertinent  to  the  topic. 

The  major  sections  of  the  report  deal  with  (a)  the  instructional  system  as  a  concept: 
(b)  the  research  evidence  bedrihg  on  the  major  system  functions,  and  (c)  methods  for  designing 
and  evaluating  the  system  in  terms  of  oost  and  effectiveness. 

Findings  and  Conclusions 

Selected  findings  and  conclusions  are  listed  below.  While  these  general  guidelines  will, 
in  somt  instances,  nsed  to  bo  supplsawntsd  by  specific  procedures  dsvised  to  most  particular 
training  situations,  they  should  serve  to  indicate  some  of  tha  considerations  involved. 

(1)  The  principal  concept  is  that  of  ths  instructional  system  as  an  intsgratsd  sst  of 
media,  equipment,  methods,  and  personnel,  all  performing  efficiently  the  functions  required  to 
accomplish  ons  or  mors  training  objectives. 

(2)  The  critical  aspects  of  practice  of  performance  art: 

(a)  To  simulate  tha  job  task,  using  a  dstailsd  task  description  as  a  quids. 

(b)  To  provide  for  knowledge  of  results. 
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(c)  To  arrange  a  suitable  practice  schedule. 

(d)  To  maximize  transfer  of  training. 

(3)  The  critical  aspects  Of  practice  of  knowledge  are: 

(a)  To  determine,  through  analysis,  the  relation  between  the  cues  and  responses 
required  by  the  knowledge. 

(b)  To  develop,  through  practice  with  knowledge  of  results,  a  high  level  of 
achievement. 

(c)  To  devise  ways  of  making  material  meaningful  to  the  trainee. 

f4)  Presentation  of  knowledge  can  be  done  successfully  by  any  of  several  meth¬ 
ods,  provided: 

(a)  The,  presentation  communicates  to  the  student. 

(b)  The  material  presented  is:  meaningful. 

(c)  The  special  characteristics  of  the  various  media  are  taken  into  account. 

(5)  The  management  of  students  is  concerned  with: 

(a)  Arrangements  for  de-ding  effectively  with  individual  differences. 

(b)  Reinforcement  of  the  learning  of  tasks. 

(c)  Overall  sequencing  of  tasks  to  be  learned. 

(d)  Controlling  avoidable  absences  from  classes. 

(6)  Automated  instruction  makes  possible  the  consistent  teaching  of  large  numbers  of 
students.  It  can  be  especially  useful  where  instructor  shortages  exist  or  when  training  must  be 
decentralized.  It  has  the  drawback  of  not  being  easily  modifiable  in  the  light  of  changing 
training  requirements. 

(7)  Instructional  devices  and  media  should  be  selected  in  terms  of  cost  and  effectiveness. 

(8)  The  entire  system  should  be  evaluated  in  terms  of  cost  and  effectiveness. 
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INTRODUCTION 


A  course  of  instruction  may  involve  any  of  a  number  of  aids  and  activities: 
lectures;  formal  and  informal  tests;  training  films  and  other  training  aids, 
devices,  and  simulators;  practical  exercises;  procedures  for  maintaining  class¬ 
room  discipline;  counseling  procedures;  programed  instruction.  How  can  these 
be  used  effectively,  and  for  what  kinds  of  instruction  are  they  suitable?  How 
can  they  be  combined  into  a  coordinated,  integrated  system,  so  that  each 
instrument  or  technique  reinforces  the  others?  How  can  one  determine  the 
relative  merits  of  different  aids  and  techniques  for  a  given  purpose  ?  These  and 
other  questions  With  which  the  course  designer  will  be  concerned  as  he  develops 
a  training  program  are  dealt  with  in  this  report. 

The  key  concept,  discussed  in  detail  in  Part  I,  is  that  of  an  instructional 
system  as  a  coordinated  set  of  instructional  aids  and  activities  designed  to  be  as 
efficient  as  possible.  The  nature  of  an  Instructional  system,  cost-effectiveness 
ratios,  automated  instruction,  and  computer-based  instruction  are  some  of 
the  topics  considered.  Part  I  is  concerned,  too,  with  the  general  functions 
that  must  be  considered  and  designed  into  an  instructional  system.  The 
advantages  and  disadvantages  of  various  ways  to  accomplish  the  goals  of  the 
system  are  described. 

The  unifying  concept  throughout  much  of  Parti  is  that  of  transfer  of  training- 
how  to  arrange  a  series  of  training  activities  so  that  one  learning  experience 
transfers  positively  to  others. 

Specific  problems  concerning  the  design  and  evaluation  of  the  system  are 
taken  up  in  Part  II.  These  include  sequencing  instruction,  designing  lessons, 
and  managing  students. 

References  which  provide  much  of  the  basis  for  this  report  are  cited 
separately  for  each  chapter,  in  order  to  provide  the  reader  with  information 
on  available  literature  pertaining  to  a  topic  in  which  he  has  special  interest. 
Additional  references  relevant  to  the  general  chapter  topics  are  also  supplied. 

This  report  is  the  third  in  a  series  of  reports  on  training  technology  by  the 
present  author.  The  other  reports  concern  training  objectives  (1)  and  quality 
control  for  training  programs  (2). 


Part  I 

INSTRUCTIONAL  SYSTEMS-CONCEPTS 
AND  FUNCTIONS 


Chapter! 

WHAT  AN  INSTRUCTIONAL  SYSTEM  IS 

For  the  purposes  of  this  report,  an  instructional  system  is  defined  as  an 
integrated  set  of  media,  equipment,  methods,  and  personnel  performing  effi¬ 
ciently  the  functions  required  to  accomplish  one  or  more  training  objectives. 
These  objectives  are  statements  of  the  performances  required  of  the  Student 
after  training. 

Modern  training  concepts  emphasize  the  importance  of  planning  training 
programs  with  a  clear  specification  of  objectives  derived  from  careful  analysis 
of  the  job  the  student  will  perform  after  graduation  (Smith,  1).  The  instructional 
system  components  required  to  reach  a  given  objective  must  be  carefully  selected 
and  assembled.  Both  the  effectiveness  of  the  components  and  the  cost  at  which 
a  given  level  of  effectiveness  can  be  attained  must  be  considered  and  measured. 

A  MODEL  OF  AN  INSTRUCTIONAL  SYSTEM 

A  general  model  of  an  instructional  system  is  presented  in  Figure  1.  The 
system  is  composed  of  a  student  and  the  following  functions:  Practice  of  Per¬ 
formance,  Practice  of  Knowledge,  Presentation  of  Knowledge,  Management  of 
Students,  and  Quality  Control.  Training  objectives  based  on  the  job  the  student 
will  perform  are  an  integral  part  of  the  system.  Figure  1  also  shows  how  the 

A  Modal  of  an  Instructional  System 


functions  work  together  to  accomplish  the  objectives  necessary  for  performing* 
the  job.  The  salient  features  of  each  art  of  the  model  are  discussed  in  this 
section.  More  detailed  research  findings  concerning  the  design  of  each  Of  the 
functions  are  described  in  later  chapters. 

The  Practice  of  Performance 

The  objectives  of  training  specify  the  tasks  to  be  learned,  as  well  as  the 
skill  and  knowledge  components  that  contribute  to  task  performance.  By  the 
practice  of  performance  is  meant  the  practice  of  the  task  and  its  skill  components. 

In  developing  the  training  objectives,  detailed  descriptions  of  each  task 
should  be  prepared  (Smith,  1).  These  descriptions  form  the  basis  for  the  design 
of  situations  in  which  the  student  practices  the  task  until  he  has  learned  it. 

To  a  certain  extent,  — T'  practice  exercise  is  a  simulation  of  the  job.  In 
some  situations,  actual  equipment  and  material  required  on  the  job  will  be  used 
in  the  practice  exercise.  In  other  situations;  obsolete  equipment  or  training 
devices  may  be  used.  Guidance  as  to  what  should  he  simulated  will  be  found 
in  the  cues,  actions,  and  indicators  of  correct  action  listed  in  the  detailed 
task  description., 

Training  devices,  widely  used  in  training,  can  be  extremely  useful.  Many 
devices,  however,  are  more  expensive  or  less  effective  than  they  should  be. 
Excessive  "realism,"  or  the  simulation  of  more  of  the  actual  equipment  than  is 
required  for  the  task,  can  be  very  costly.  However,  leaving  out  relevant  aspects 
of  the  task  can  reduce  effectiveness. 

One  distinction  that  has  been  made  with  regard  to  training  devices  is  that 
between  "whole  task"  and  "part  task"  devices.  The  "whole  task"  device,  which 
provides  practice  on  one  or  more  complete  tasks,  is  frequently  very  expensive. 
Under  some  circumstances  "part  task"  devices,  which  provide  practice  on  only 
a  key  portion  of  the  task  or  set  of  tasks  practiced  on  the  "whole  task"  trainer, 
can  provide  effective  training  at  a  much  lower  cost. 

Certain  tasks  may  be  taught  using  obsolete,  rather  than  the  newest,  equip¬ 
ment.  This  practice,  where  appropriate,  may  reduce  costs.  Again,  the  detailed 
task  description  is  the  guide. 

Practice  exercises  and  training  devices  should  be  designed  in  such  a  way 
that  appropriate  forms  of  knowledge  of  results  can  be  provided  the  students. 
The  nature  of  the  knowledge  of  results  must  be  suited  to  the  student's  needs  or 
it  will  be  ineffective.* 

Properly  designed  practice  exercises  provide  transfer  of  training  to  the  job. 
That  is,  what  is  learned  in  the  practice  situation  is  transferred  to  a  new  situation— 
the  job— so  that  learning  time  in  the  job  situation  is  reduced.  In  situations 
j.n  which  a  task  must  eventually  be  practiced  on  the  actual  equipment  or  on  a 
complex  simulator,  the  use  of  simpler  devices  may  provide  transfer  to  the  com¬ 
plex  simulator  and  thus  reduce  the  requirements  for  supplies  of  the  more  expen¬ 
sive  equipment.  In  this  way  the  overall’ cost  of  effective  training  may  be  reduced. 

The  Practice  of  Knowledge 

The  learning  of  every  task  presupposes  the  learning  of  some,  even  if  only  a 
few,  knowledge  components.  Knowledge  refer?  to  symbolic  processes.  The 
practice  of  knowledge  means  that  the  student  gives  symbolic  responses  and 
receives  information  as  to  their  correctness. 


'Knowledge  of  results  is  discussed  in  detail  in  (chapter  2. 


The  knowledge  components  are  required  for  effective  task  performance,  and 
mastering  them  will  transfer  to  the  practice  of  performance.  The  equipment  and 
material  for  the  practice  of  knowledge  are  frequently  less  expensive  than  those 
for  practicing  performance,  and  it  is  efficient  to  take  advantage  of  this.  At  the 
same  time,  research  results  show  that  little  transfer  takes  place  unless  the 
knowledge  is  mastered  by  the  student.  '), 

The  guide-  for  the  particular  knowledges  that  should  be  practiced  for  each 
task  or  set  of  tasks  is  the  hierarchical  organization  of  the  knowledge  components 
of  that  task.  To  design  effective  procedures  for  practice,  special  analyses  of 
these  knowledge  components  must  be  made. 

Presentation  of  Knowledge 

Much,  if  not  most,  training  activity  involves  the  presentation  of  knowledge  to 
the  student,  often  with  little  else  provided.  Presentation  can  employ  a  variety 
of  techniques— lectures,  reading,  graphic  aids,  films,  television,  and  tape 
recorders,  for  example.  A  great  deal  of  knowledge  can  be  presented  in  a  short 
time.-  A  single  presentation  can.  be  received  by  many  students  at  the  same  time. 
Presentation  is  less  expensive  than  practice. 

These  seeming  advantages  are  often  illusory.  Students  frequently  fail  to 
pay  attention.  Presentations  are  often  made  at  such  an  abstract  level,  or  with 
such  speed,  that  the  student  fails  to  grasp  the  content.  Or  the  level  of  presen¬ 
tation  may  be  too  elementary,  and  the  student  becomes  bored. 

In  a  group  of  students,  the  majority  will  learn  something  from  a  well- 
organized,  properly  pa^ced  presentation  geared  to  the  level  of  the  student.  How¬ 
ever,  the  research  evidence  is  quite  clear  that  students  learn  more  when 
presentation  is  integrated  with  practice.  Since  presentation  is  a  one-way  street, 
there  is  no  way  for  the  training  system  to  determine  the  effectiveness  of 
presentation  until  the  practice  stage  has  been  reached. 

Management  of  Students 

The  function  of  management  of  students  refers  to  those  features  of  the  sys¬ 
tem  that  keep  the  student  productively  participating  in  the  learning  activity. 
These  include: 

(1)  Short-term  incentives  to  insure  motivation 

(2)  Arrangements  for  dealing  with  individual  differences 

(3)  Procedures  for  minimizing  absences  from  class 

In  the  design  of  the  student  management  function,  incentives  contingent  on 
successful  progress  in  training  can  be  provided,  although  this  is  frequently  not 
done.  It  is  also  possible  to  avoid  the  kinds  of  practices  that  would  amount  to 
rewarding  students  for  failure. 

A  number  of  possibilities  exist  for  dealing  with  differences  in  student 
ability.  All  of  these  have  in  common  some  form  of  individualized  treatment  of 
the  student.  These  methods  create  administrative  problems  for  most  Army 
training  programs,  which,  on  paper  at  least,  are  generally  the  same  length  for 
every  student.  One  way  of  adapting  to  individual  differences  is  by  recycling,  to 
create  courses  that  are  really  of  variable  length. 

A  set  of  procedures  should  be  established  to  monitor  student  progress  and— 
depending  on  the  progress  or  lack  of  it— provide  the  basis  for  making  various 
decisions  concerning  the  student. 
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Quality  Control 

The  purposes  of  a  quality  control  system  are  described  in  detail  in  an 
earlier  report  (Smith,  2).  Generally,  such  a  system  is  a  set  of  procedures 
involving  the  development  and  use  of  tests  of  the  objectives.  The  tests  are 
administered  to  the  students,  and  the  results  are  used  as  a  basis  for  continu  ng 
improvement  of  the  instruction. 

Quality  control  has  a  very  important  function  in  the  development  of  an 
instructional  system.  This  is  to  obtain  data  on  the  effectiveness  of  the  system 
in  accomplishing  its  objectives  and  to  provide  guidance  as  to  how  to  change  the 
instructional  system  to  increase  its  effectiveness. 

INSTRUCTIONAL  SYSTEM  DESIGN 

The  major  steps  in  the  design  of  the  instructional  system  are: 

(1)  Preparing  the  training  objectives 

The  training  objectives  define  the  purposes  of  the  instructional 
system.  They  should  include  not  only  a  statement  of  the  tasks  to  be  performed 
on  a  job  at  the  end  of  training,  but  also  the  component  knowledges  and  skills 
required  for  task  performance.  A  properly  prepared  set  of  objectives  will  pro¬ 
vide  overall  guidance  tp  the  system  designer,  which  Will  aid  in  the  integration 
of  the  various  parts  of  the  system. 

(2)  Sequencing  the  objectives  of  the  system 

The  objectives  of  the  instructional  system  cannot  be  achieved  all 
at  once;  they  must  be  sequenced.’  Proper  sequencing  of  training  activities  is 
vital  to  the  effectiveness  of  the  system.  Some  forms  of  sequencing  make  the 
objectives  interesting  and  easy  to  master;  others  make  them  boring  and  diffi¬ 
cult  to  master. 

(2)  Identifying  required  functions 

The  model  of  an  instructional  system  (Figure  1)  suggests  the  func¬ 
tions  required  by  the  system.  For  any  particular  objective,  the  relative  empha¬ 
sis  on  any  particular  function  may  differ  from  that  of  another  objective. 

(4)  Selecting  components  and  procedures 

After  deciding  generally  on  the  relative  emphasis  to  be  given  each 
function  for  any  particular  objective,  system  components  and  procedures  are 
selected  to  accomplish  the  objective.  How  is  the  task  to  be  practiced?  Using 
actual  equipment,  training  devices  (what  kinds?),  obsolete  equipment,  or  what ? 
What  kinds  of  materials  must  be  prepared  to  provide  practice  of  knowledge  ? 
What  methods  for  the  presentation  of  knowledge  will  be  used— books,  lectures, 
films,  television? 

(5)  Analyzing  cost-effectiveness 

A  cost-effectiveness  analysis  should  be  made  of  the  system  possi¬ 
bilities.  If  several  components  and  procedures  seem  equally  effective,  their 
relative  costs  should  serve  as  a  guide  to  their  selection.  Analysis  of  costs  and 
effectiveness  is  a  continuing  process  throughout  the  development  and  operation 
of  the  system.  i 

(6)  Coordinating  components  and  procedures 

The  entire  instructional  system  should  be  examined  to  determine 
whether  it  will  be  a  smoothly  working,  integrated  whole.  This  examination 
should  be  continuous,  during  both,  the  later  stages  of  system  development 
and  the  actual  system  operation.  Flow  charts  and  computer  simulation  are 

‘Guidance  for  sequencing  is  presented  in  later  chapters  of  this  report. 
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valuable  tools  for  this  type  of  study;  observation  of  the  operation  of  the  system 
is  indispensable. 

(7)  Evaluating  the  system 

The  need  for  coat -effectiveness  concepts  in  selecting  system 
components  has  been  mentioned.  It  is  also  desirable  to  evaluate  the  complete 
system  in  terms  of  cost  and  effectiveness.  The  index  of  effectiveness  is  the 
number  of  attainable  objectives  actually  attained  by  the  students.  This  informa¬ 
tion  will  come  from  the  quality  control  system.  The  total  cost  must  be  devel¬ 
oped  from  data  on  the  variety  of  costs  that  make  up  the  total. 

System  evaluation  is  a  continuous  process.  Systems  are  not  static;  they 
develop  and  change.  The  impact  of  external  pressures  may  degrade  the  sys¬ 
tem.  Improvements  of  different  kinds  wiil  be  made.  It  is  desirable  to  know 
the  impact  of  these  pressures,  and  the  effect  of  improvements  on  the  cost- 
effectiveness  ratio. 


Chapter  2 

THE  PRACTICE  OF  PERFORMANCE 


The  culmination  of  any  training  sequence  is  student  practice  of  the  per¬ 
formance  required  by  the  training  objective.  As  used  in  this  chapter.,  the  term 
performance  refers  to  both  the  task  level  objective  and  the  skill  (not  knowledge) 
components  of  the  task. 

Factors  that  contribute  to  the  effectiveness  with  which  performance  is 
practiced,  as  well  as  those  that  can  reduce  the  cost  of  practice,  are  described 
in  this  chapter.  Also,  general  guidance  is  provided  to  assist  the  instructional 
system  designer  in  designing  effective  practice  situations  at  minimum  cost. 
This  guidance  leaves  ample  room  for  the  exercise  of  creative  ingenuity  on  the 
part  of  the  system  designer. 

PRACTICE  OF  THE  TASK 

A  task  is  a  meaningful  unit  of  job  performance.  It  is  a  group  of  activities 
that  generally  occur  close  together  and  have  a  common  purpose. 

It  is  essential  that  the  student,  when  he  practices  performance,  practice 
the  task  for  which  he  is  being  trained.  And  he  should  practice  the  whole  task, 
not  just  a  part  of  it.  As  part  of  the  process  of  developing  training  objectives, 
preparation  of  a  detailed  task  description— a  complete  statement  of  the 
cues,  actions,  and  indicators  of  correct  action  to  be  found  in  the  task— is 
recommended  (1).  The  best  insurance  that  the  whole  task  will  be  learned  is  to 
develop  a  practice  situation  based  on  this  detailed  task  description. 

The  practice  situation  should,  then,  provide  the  cues  to  which  the  trainee 
should  respond  and  controls  or  other  equipment  on  which  he  should  act;  it 
should  also  provide  him  with  indications  as  to  whether  his  action  was  correct. 
The  sequence  of  actions  should  be  indicated  in  the  task  description;  the  student 
should  practice  this  sequence,  and  no  other— unless,  of  course,  alternate  pro¬ 
cedures  are  stated  in  the  task  description. 

Thus,  the  practice  situation  represents  a  simulation  of  the  essential  ele¬ 
ments  of  the  job  task— cues,  actions;  and  indications  of  correct  action.  From 
the  standpoint  of  costs,  it  is  generally  expensive  and  unnecessary  to  simulate 
additional  features  of  the  job  situation. 

KNOWLEDGE  OF  RESULTS  (KOR) 

Forms  of  KOR 

Knowledge  of  results  (KOR)  can  take  many  forms.  It  can  arise  naturally 
out  of  the  task  environment.  It  can  come  from  another  person,  such  as  a  stu¬ 
dent  or  instructor.  Special  signals,  such  as  lights  or  buzzers,  may  be  used. 
Te3t  scores  may  be  reported  to  the  student.  The  purpose  of  providing  KOR  is 
to  inform  the  student  whether  his  action  was  correct,  or,  in  the  case  of  con¬ 
tinuous  tasks,  whether  his  level  of  accuracy  is  satisfactory.  Certain  types  of 
KOR  are  particularly  significant. 


12 


Natural  KOR.  Natural  KOR  arises  out  of  the  task  environment.  In  the 
detailed  task  description  it  is  the  indication  of  correct  action.  In  some  instances, 
it  is  the  evidence  of  the  student’s  own  senses.  In  other  instances,  the  task  is  so 
unfamiliar  that  the  student  must  learn  the  indications  of  correct  action  that 
represent  KOR,  if  they  are  present.  Once  he  has  learned  these  indications, 
however,  they  are  clear  signals  to  him.  In  still  other  situations,  the  indications 
that  form  KOR  are  unclear  even  after  considerable  training. 

The  clarity  of  the  natural  KOR  is  an  important  consideration,  because 
it  interacts  with  the  effectiveness  of  artificial  KOR. 

Artificial  KOR.  Artificial  KOR  does  not  arise  out  of  the  task  itself,  but  is 
provided  by  additional  means,  such  a3  instructor  critiques  of  student  perform¬ 
ance,  and  mechanical  aids  such  as  buzzers,  lights,  or  clicks. 

Artificial  KOR  should  be  employed  with  reservations,  as  it  may  itself 
affect  performance.  If  it  is  provided  during  practice  of  the  task,  it  may  raise 
the  level  of  performance  during  practice;  however,  since  the  KOR  is  not  natural 
to  the  task,  performance  may  drop  when  the  student  goes  to  the  job  and  the  arti¬ 
ficial  KOR  is  no  longer  present.  A  number  of  tracking  experiments  have  shown 
that  the  artificial  KOR  raises  the  level  of  performance,  but  that  perfcrmance 
drops  when  it  is  withdrawn  (Bilodf.au,  3;  Bilodeau,  Bilodeau,  and  Schumsky,  4; 
Goldstein  and  Rittenhouse,  5).  This  finding  is  not  a  universal  one,  however. 
Other  studies  have  shown  no  reduction  in  performance  when  artificial  KOR  was 
withdrawn.  Kinkade  (6)  found  that  artificial  KOR  favorably  affected  learning  on 
a  permanent  basis  when  there  was  clearly  perceived  natural  KOR,  even  after 
the  artificial  KOR  was  withdrawn.  On  the  other  hand,  when  natural  KOR  was 
not  clearly  perceived,  performance  level  was  reduced  when  artificial  KOR 
was  withdrawn. 

A  number  of  experiments  indicate  that  artificial  KOR  may  be  added 
when  natural  KOR  is  clearly  perceived  by  the  student,  but  should  not  be  added 
when  natural  KOR  is  unclear. 

Performance  KOR  vs  Learning  KOR.  Miller  (7)  draws  a  useful  distinction 
between  performance  feedback  and  learning  feedback  (feedback  is  a  co-  tmouly 
used  alternative  term  for  KOR). 

Performance  feedback,  or  KOR,  is  the  ertvironmental  information  that 
tells  us  to  change  pur  next  response  so  as  to  perform  the  task  better.  Suppose 
we  want  to  drive  quickly  from  home  to  the  office.  We  don't  want  to  risk  getting 
a  ticket  for  speeding,  so  we  look  from  time  to  time  at  our  speedometer.  If  we 
are  too  far  under  the  speed  limit,  we  push,  down  oh  the  accelerator;  if  we  are 
above  the  speed  limit,  we  let  up  on  the  accelerator.  This  is  performance  KOR, 
because  it  permits  us  to  change  our  responses  to  perform  an  ongoing  task 
more  effectively. 

Learning  KOR  is  given  after  a  response  has  been  made,  and  provides 
a  basis  for  a  more  correct  response  to  the  same  cue  conditions  when  they  next 
occur.  It  is  as  if  the  individual  said  to  himself,  "I  now  see  what  I  should 
have  done." 

The  significance  of  the  distinction  between  learning  KOR  and  pe*  rom¬ 
ance  KOR  is  this:  Instances  showing  poor  performance  after  the  withdrawal 
of  artificial.  KOR  have  been  examples  of  artificial  performance  KOR,  rather 
than  artificial  learning  KOR. 

Generalizations  Concerning  KOR 

Ammons  (8)  reviewed  the  literature  and  developed  a  number  of  generaliza¬ 
tions  and  possible  applications  of  KOR*  Some  will  be  presented  here,  as  a  guide 
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to  making  effective  use  of  KOR.  These  generalizations  should  not  be  accepted 
uncritically,  however;  they  may  require  checking  in  particular  settings. 

(1)  The  performer  usually  has  hypotheses  (or  guesses)  about  what  he 
should  do  and  how  he  should  do  it.  These  interact  with  KOR.  The  guesses  the 
student  makes  about  the  practice  situation  may  make  him  pay  attention  to  the 
wrong  cues.  Or  they  may  lead  to  incorrect  responses.  The  student  may  use 
incorrect  or  irrelevant  information  as  KOR.  It  is  therefore  important  to  cor¬ 
rect  wrong  guesses  as  soon  as  possible. 

(2)  Some  KOR  is  nearly  always  available  to  the  human  performer.  The 
learner  uses  what  information  is  available  to  him,  from  the  environment  or  from 
his  own  memory.  In  a  number  of  studies  cited  by  Ammons,  groups  not  receiving 
KOR  that  was  designed  by  the  experimenter  learned  as  well  as  those  receiving 
KOR  in  a  formally  designed  way.  The  groups  not  receiving  formal  KOR  were 
getting  their  own  KOR  from  the  environment.  However,  training  inefficiency 
occurs  when  the  KOR  obtained  from  the  environment  is  irrelevant  or  wrong. 

(3)  KOR  increases  the  rate  of  learning  and  the  level  of  performance  reached 
by  learning.  In  the  great  majority  of  instances,  an  experimental  group  that  has 
received  KOR  learns  more  quickly  and  reaches  a  higher  level  ot  performance 
than  one  that  has  not. 

(4)  KOR  affects  motivation.  There  is  a  tendency  for  KOR  to  make  learning 
more  interesting  to  the  student.  However,  he  may  be  interested  simply  in  scor¬ 
ing  higher,  and  he  may  cheat  or  take  advantage  of  weaknesses  in  the  situation 
to  make  a  high  score.  It  is  important  that  practice  situations  be  designed  to 
control  this  possibility. 

(5)  The  more  specific  the  KOR,  the  faster  the  improvement,  and  the  higher 
the  performance  level.  Other  things  being  equal,  the  more  specific  the  informa¬ 
tion  as  to  hov/  the  student  lias  performed;  the  more  likely  he  will  be  to  correct 
erroneous  responses.  This  statement  should  be  qualified,  however.  In  the 
early  stages  of  practice  on  a  complex  task,  too  much  specific  information  may 
be  confusing.  In  fact,  one  of  the  things  a  student  has  to  learn  is  how  to  make  use 
of  a  wider  range  of  information. 

(6)  The  longer  the  delay  in  providing  KOR  to  the  student,  the  smaller  effect 
it  has.  Most  studies  show  that  delays  of  as  much  as  a  few  seconds  after  the 
response  can  make  KOR  less  effective.  A  study  by  Bilodeau  and  Bilodeau  (9) 
shows  that  if  there  is  no  activity  required  of  the  student  between  the  response 
and  KOR,  delay  of  KOR  has  no  effect.  However,  in  the  majority  of  learning 
situations  further  activity  will  be  required. 

Ir.  most  practical  learning  situation^,  this  principle  of  providing  KOR 
rapidly  is  violated.  Tests  are  scored  but  the  results  are  not  given  to  the  student 
for  several  days,  for  example. 

(7)  When  KOR  is  decreased,  performance  level  tends  to  drop.  While  pro¬ 
vision  of  artificial  KOR  may  raise  the  level  of  performance  during  practice, 
performance  tends  to  drop  when  the  artificial  KOR  is  withdrawn  if  there  is  no 
clearly  perceived  natural  KOR, 

Suggestions  for  Application  of  KOR 

Most  of  the  following  suggestions  for  applying  what  is  known  about  KOR  are 
taken  from  Ammons '  (!)  review,  with  modifications: 

(1)  Develop  methods  for  correcting  or  eliminating  student  hypotheses 
or  guesses  that  conflict  with  KOR.  Ignorance  on  the  part  of  students  as  to  what 
must  be  learned  can  lead  to  rumors  and  interfering  hypotheses  about  what  is  to 
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be  learned.  One  way  of  correcting  this  situation  is  to  provide  students  with  a 
set  of  training  objectives.  Another  is  to  provide  an  accurate  description  of  the 
practice  situation,  and  any  training  devices  used  in  it. 

(2)  The  design  of  all  training  and  operational  equipment  should  provide 
for  KOR.  If  KOR  is  designed  into  operational  equipment  it  becomes  natural  KOR, 
and  can  be  used  in  the  practice  situation  without  any  concern  over  its  withdrawal 
in  the  actual  job. 

Care  should  be  taken  that  natural  KOR  can  be  clearly  perceived 
by  the  student  if  artificial  KOR  is  to  be  used. 

Consideration  should  also  be  given  to  planning  the  amount  and 
specificity  of  KOR  to  be  given  at  different  stages  of  practice.  Early  in  practice 
of  a  complex  task,  too  much  detail  can  be  confusing.  As  practice  proceeds, 
however,  the  student  should  be  able  to  use  more  complex  information, 

(3)  An  appropriate  lag  in  KOR  should  be  provided  for  in  the  design  of 
training  and  operational  equipment.  It  is  important  that  learning  KOR  follow 
practice  performance  of  the  task  by  an  appropriate,  but  short  interval.  The 
optimum  interval  for  a  given  task  must  often  be  found  by  experimentation. 

(4)  If  it  is  impossible  to  provide  KOR  all  the  time,  give  it  as  often  as 
possible.  A  further  qualification  on  this  statement  is  to  avoid  long  sequences 
without  KOR.  Instead,  in  a  sequence  including  practice  trials  with  KOR,  practice 
trials  wimout  KOR  should  be  interspersed  randomly. 

(5)  If  a  delay  in  KOR  must  occur,  the  student  should  wait  quietly  until. 
KOR  is  received.  Previously,  evidence  was  cited  showing  that  a  delay  in  KOR 
did  hot  reduce  performance  as  long  as  there  was  no  formal  requirement  for 
student  activity  between  response  and  KOR. 

(6)  Where  KOR  must  show  the  direction  of  the  actual  student  response 
in  relation  to  the  correct  response,  include  specific,  details.  If  the  student 
responds  by  overshooting  or  undershooting,  or  aiming  too  far  to  the  left  or 
right,  it  is  an  example  of  a  situation  in  which  the  direction  of  the  response 
should  ';e  shown  to  students  in  the  KOR.  The  inclusion  of  a  directional  com¬ 
ponent  will  be  more  effective  than  simply  telling  the  student  he  was  off-target  or 
on-target.  However,  the  student  shouldn't  be  overloaded  with  too  much  com¬ 
plex  detail  in  early  stages  of  practice.  It  might  be  best  to  just  indicate  to  him 
that  he  was,  for  example,  overshooting  or  too  far  to  the  left.  Then,  as  the  train¬ 
ing  progresses,  increasingly  specific  details  concerning  direction  and  distance 
may  be  included  in  KOR. 


TRANSFER  OF  TRAINING 

A  student  studies  a  photograph  of  a  piece  of  equipment.  He  learns  the 
appearance  and  location  of-atiEn  significant  part  of  the  equipment.  Later,  he 
mentally  practices  the  steps  of  the  procedure,  referring  to  the  photograph. 
Then  he  practices  the  procedure  on  the  actual,  equipment.  Because  of  his  pre¬ 
liminary  work  on  the  photograph,  he  is  able  to  learn  on  the  actual  equipment 
much  faster  than  if  he  had  not  engaged  in  the  earlier  pr  ictice.  What  he  learned 
in  the  earlier  work  has  transferred  to  learning  on  the  actual  equipment. 

By  transfer  of  training  is  r.veant  that  what  a  Student  learns  in  one  situation 
affects  the  ease  of  learning  in  another  situation.  This  effect  may  be  positive 
or  negative.  Proper  management  of  training  activities  to  enhance  positive 
transfer  contributes  significantly  to  the  effectiveness  and  economy  of  training. 
Positive  transfer,  for  instance,  is  the  basis  for  the  design  and  t'/se  of 
training  devices. 
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identification  of  Transferable  Content 

Identification  of  task  component  knowledges  and  skills  and  synthesis  of  the 
components  into  an  organized  structure  are  part  of  the  process  of  developing 
training  objectives  (1).  These  steps  are  prerequisites  to  identification  of  spe¬ 
cific  knowledges  and  skills  that  will  transfer  to  the  performance  of  the  total  task. 

The  organization  of  the  component  skills  and  knowledges  will  serve  as  a 
guide  to  the  sequencing  of  content  to  encourage  positive  transfer.  For  instance, 
by  mastering  the  lower  level  components  of  the  task,  practice  on  the  total  task 
may  be  reduced.  There  are  a  few  exceptions,  which  will  be  treated  in  later 
sections  of  this  chapter.  In  the  following  sections,  research  findings  that  offer 
guidance  „n  planning  for  positive  transfer  will  be  described,  as  well  as  some 
precautions  that  suggest  limitations  on  the  concept  of  transfer. 

Transfer  of  training  has  major  implications  for  the  cost  of  training.  As 
will  be  shown  in  the  later  discussions  of  training  devices  and  the  practice  of 
knowledge,  students  can  practice  on  inexpensive  devices;  this  practice  will  then 
transfer  to  more  expensive  practice  situations.  The  inexpensive  situation  may 
be  either  a  complete  or  a  partial  substitute  for  the  more  expensive  situation. 

Component  Practice  and  Transfer 

In  general,  if  a  task  has  been  correctly  analyzed  into  its  component  knowl¬ 
edges  and  skills,  practice  on  a  component  will  transfer  to  the  task.  The  situa¬ 
tion  with  regard  to  knowledge  will  be  discussed  in  the  next  chapter. 

Several  studies  (Bilodeau,  10;  North  and  Harrington,  1 1;  Seymour,  12)  have 
shown  that  component  skills  can  be  practiced  separately  from  the  total  task  with 
considerable  transfer.  However,  there  are  two  exceptions  that  must  be  con¬ 
sidered  when  planning  to  use  component  practice. 

The  first  exception  is  time-sharing.  A  continuous  task,  such  as  flying  an 
airplane  or  driving  a  vehicle,  often  must  be  performed  during  the  same  time 
period  as  a  procedural  task.  Arrangements  must  be  made  to  practice  these 
together  if  satisfactory  proficiency  is  to  be  attained  (Adams  and  Hufford,  13; 
Briggs  and  Naylor,  14;  Daugherty,  Houston,  and  Nicklas,  15). 

The  second  exception  occurs  when  there  is  interaction  between  the  task 
components  (Briggs  and  Waters,  16).  For  example,  in  making  a  turn  in  an  air¬ 
craft,  the  turn  controls  are  operated,  but  at  the  same  time  the  aircraft  must  be 
banked  to  an  appropriate  degree.  If  turning  and  banking  are  practiced  as  sep¬ 
arate  components,  there  will  be  little  transfer  to  the  whole  task,  which  involves 
both  banking  and  turning. 

In  summary,  if  the  total  task  involves  either  time-sharing  of  activities  or 
interaction  of  skill  components,  do  not  expect  much  transfer  from  component 
practice  to  the  practice  of  the  entire  task. 

EFFECTIVE  PRACTICE 

A  number  of  factors  that  contribute  to  effective  or  efficient  practice  are 
described  in  this  section. 

Distribution  of  Practice  and  Rest 

Distribution  of  practice  and  rest  periods  affects  the  performance  being 
practiced.  If  the  schedules  for  rest  and  practice  are  such  that  they  depress 
performance,  then  we  may  be  extending  practice  far  beyond  the  amount  really 
necessary  to  develop  an  appropriate  degree  of  proficiency. 
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A  trial  is  defined  as  a  continuous  practice  session,  whether  the  task  is 
practiced  once  or  several  times  (in  the  case  of  short  tasks).  Typically,  a  trial 
will  be  followed  by  a  short  rest. 

It  is  difficult  to  generalize  from  the  studies  that  have  been  performed  to  an 
optimum  practice  schedule;  however,  the  following  suggestions  should  help  to 
prevent  serious  procedural  errors. 

When  trials  are  short— on  the  order  of  a  few  minutes  or  less— then  a  rest  of  30  to 
45  seconds  should  be  allowed  between  trials,  followed  by  aten-minute  break  after 
every  ten  trials.  Iftrials  arelonger,  then  a  rest  period  of  35  to45  seconds  should 
be  allowed  between  trials,  followed  by  a  ten-minute  break  after  every  few  trials. 

If  performance  on  the  task  is  being  measured  to  determine  when  a  student 
has  attained  satisfactory  proficiency,  it  is  suggested  that  measurements  be 
taken  immediately  after  the  ten-minute  break. 

If  the  student  is  given  little  or  no  rest,  his  performance  level  will  be  lowered. 
This  will  mean  he  will  have  to  practice  much  longer  than  necessary  to  meet 
the  appropriate  standard  of  proficiency.  On  the  other  hand,  if  rest  periods  are 
excessively  long,  time  will  be  wasted. 

It  is  assumed  here  that  the  task  makes  minimal  demands  on  the  student's 
physical  strength  hnd  endurance.  If  practicing  the  task  develops  significant 
amounts  of  fatigue,  then  longer  rest  periods  must  be  prpvided. 

Studies  relevant  to  this  section  are  those  of  Adams  and  Reynolds  (17), 
Norris  (18),  and  Reynolds  and  Bilodeau  (19). 

Verbalization  During  Practice 

In  the  practice  of  procedures  arid  problem-solving  tasks,  it  has  been  found 
advantageous  to  require  the  student  to  talk  his  way  through  the  task  while  prac¬ 
ticing  (Esper  and  Lovaas,  20;  Gagn£  and  Smith,  21;  Newman,  22;  and  Ray,  23). 
This  is  one  of  the  features  of  the  LOCKON  method  of  instruction  (Woolman,  24), 
developed  by  HumRRO  for  the  training  of  Nike  missile  operators. 

Pacing 

Some  tasks  are  externally  paced.  That  is,  the  actions  must  be  performed 
within  time  limits  established  by  a  sequence  of  cues  that  are  also  timed.  If  the 
task  is  externally  paced,  practice  should  also  be  externally  paced.  The  research 
evidence  shows  some  trend  toward  an  advantage  for  groups  practicing  on 
externally  paced  tasks  when  they  switch  to  self-paced  conditions  (Adams,  25; 
Anderson,  Kresse,  and  Grant,  28;°Nystrom,  Morin,  and  Grant,  27  and  28). 

Overlearning 

It  is  generally  desirable,  especially  if  retention  of  a  skill  is  particularly 
important,  for  the  student  to  overlearn.  Overlearning,  in  the  sense  used  here, 
can  be  defined  as  additional  practice  after  performance  standards  have  been 
met  by  the  student.  Important  tasks  should  be  overlearned,  especially  if  there 
is  little  expectation  of  frequent  review  after  initial  learning.  Overlearning,  in 
addition  to  aiding  retention,  tends  to  prevent  skill  from  deteriorating  under 
stressful  conditions,  such  as  combat  or  other  emergencies. 

Mental  Practice 

Mental  practice  involves  the  individual  thinking  his  way  through  the  task, 
and  perhaps  making  appropriate  motions  even  though  the  normal  job  cues  are 
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not  present.  Shadow-boxing  is  an  example  of  such  practice.  If  practice  is 
required  when  actual  equipment  or  training  devices  are  not  available,  mental 
practice  might  be  considered.  In  one  situation,  the  practice  of  the  one-hand 
basketball  foul  shot  (Clark,  29),  it  was  nearly  as  effective  as  physical  practice. 

Preparation  for  Practice 

A  situation  frequently  arises  in  practical  exercises  that  may  be  a  major 
contributor  to  inefficiency.  This  is  the  requirement  for  the  student  to  spend 
considerable  time  preparing  his  materials  before  he  can  practice.  Breadboards 
of  electrical  components  may  have  to  be  assembled  before  the  student  can 
practice;  laboratory  equipment  may  have  to  be  set  up;  equipment  may  have  to 
be  cheeked  out  of  supply. 

This  preparatory  activity  is  frequently  wasted  time,  and  it  should  be 
reduced  to  an  absolute  minimum.  If  equipment  cannot  be  set  up  in  advance, 
then  the  trainee  should  have  clear  instructions  so  that  he  can  prepare  the 
materials  in  as  little  time  as  possible. 

TRAINING  DEVICES 

Some  of  the  more  important  concepts  relating  to  training  devices  are 
briefly  described  in  this  section.  The  interested  reader  is  referred  to 
Adams  (30),  Gagne  (31),  Miller  (7,  32),  and  Lumsdaine  (33,  34). 

Why  Training  Devices  ? 

Every  practice  situation  is  in  some  sense  a  simulation  of  die  job.  It  was 
pointed  out  earlier  in  this  chapter  fiat  the  task  must  be  practiced  as  defined 
by  the  cues,  actions,  and  indications  of  correct  action  found  in  the  detailed 
task  description.  Practice  of  the  task  must  provide  (a)  simulation  of  the 
environment  to  provide  the  necessary  cues,  as  inexpensively  as  possible,  (b)  an 
appropriate  control  or  tool  upon  which  the  student  can  act,  and  (c)  natural  KOR, 

In  many  circumstances  these  features  can  be  provided  using  actual  equip¬ 
ment  in  a  realistic  job  environment.  In  other  circumstances,  such  as  the  fol¬ 
lowing,  it  is  better  to  use  training  devices: 

(1)  The  training  device  is  frequently  less  expensive  thai.  the  actual 
equipment,  while  still  simulating  the  essential  elements  of  the  task.  Also,  the 
actual  equipment  may  require  the  use  of  fuel  or  additional  personnel  net  needed 
with  the  de  ice. 

(2)  The  training  device  may  be  the  only  feasible  way  to  practice  a 
task,  as  when  there  is  a  possibility  of  damage  to  equipment,  injury  to  students, 
or  destruction  of  property  in  performing  the  task. 

(3)  Training  devices  may  be  more  reliable  for  practice  purposes 
than  the  actual  equipment.  For  instance,  an  operator's  console  in  a  missile 
system  is  connected  to  other  parts  of  the  system,  and  when  the  other  parts  are 
out  of  action,  the  operator's  console  may  be  affected.  Through  simulation,  many 
such  problems  can  be  eliminated. 

Cost-Effectiveness  of  Training  Devices 

A  training  device  is  one  element  in  the  practice  situation.  It  is  rarely  the 
entire  practice  situation.  The  device  should  simulate  the  task  to  be  performed, 


including  the  cues,  actions,  and  natural  KOR  of  the  task.  All  of  the  factors 
that  govern  effective  practice  should  be  built  into  the  device,  or,  if  this  is 
impractical,  into  accompanying  lesson  plans. 

Complete  simulation  of  the  actual  equipment  is  seldom  required,  especially 
if  the  actual  equipment  contains  parts  not  germane  to  the  task.  Numerous 
research  studies  indicate  that  unnecessary  fidelity  to  the  actual  equipment  may 
introduce  material  not  relevant  to  the  particular  task  being  practiced,  and  there¬ 
fore  reduce  training  value  of  the  device.  A  study  that  illustrates  this  point  is 
that  of  Denenberg  (35),  which  compared  the  effectiveness  of  a  tank  hull  trainer, 
costing  $10,000,  and  a  mock-up  whose  materials  cost  $27.  The  mock-up  was 
superior  to  the  trainer  for  teaching  starting  and  stopping  procedures;  in  fact, 
trainees  taught  by  the  mock-up  started  and  stopped  the  M47  tank  nearly  as  well 
as  trainees  who  had  previous  experience  starting  and  stopping  a  tank.  The 
mock-up,  trainer,  and  tank  were  equally  effective  in  teaching  the  nomenclature 
and  location  of  the  driver's  instruments  and  controls. 

A  modern  vi(\v  of  the  training  device  considers  it  as  an  element  in  a  train¬ 
ing  system.  The  desired  characteristics  and  proposed  method  of  using  a 
device  should  therefore  be  considered  during  the  development  of  training.  When 
a  training  device  will  be  extremely  costly,  th*>  specifications  for  procurement  of 
the  devices  should  call  for  development  and  test  of  a  training  system  for  attain¬ 
ing  the  appropriate  objectives. 

Procedural  Trainers 

A  trainer  designed  solely  for  the  practice  of.  a  procedure— a  step-by-step 
series  of  activities  involving  no  special  skill  requirements— can  be  a  very 
rough  approximation  of  the  actual  equipment.  Photographs  or  drawings,  mounted 
on  board,  of  about  the  same  size  as  the  actual  equipment,  will  serve  inexpensively 
to  teach  many  procedures  involving  consoles  and  other  equipment  in  which  all 
cues  and  actions  are  in  plain  view.  If  feasible,  KOR  can  be  given  by  an  instructor, 
or  by  another  student  who  follows  a  description  of  the  procedure.  A  few  trials 
on  the  actual  equipment  or  a  more  realistic  trainer  may  serve  to  prepare  the 
student  for  proficiency  measurement  (Cox,  et  al.,  36). 

Skill  and  Continuous  Task  Trainers 

Continuous  tasks  usually  require  complex  displays  and  accurate  control 
display  relationships.  Continuous  task  trainers  tend  to  be  more  complex  and 
more  expensive  than  procedural  trainers.  Aircraft  simulators  and  driver 
trainers,  and  radar  tracking  simulators  are  types  of  continuous  task  trainers. 

Many  of  these  simulators  permit  the  practice  of  all  tasks  involved  in  the 
job.  Time  on  the  simulator  may  be  reduced— and  hence  fewer  complex  Simu¬ 
lators  will  be  required— if  certain  procedural  tasks  can  be  practiced  on 
simpler  devices. 

Obsolete  Equipment 

New  equipment  for  training  is  frequently  in  short  supply.  Sometimes,  how¬ 
ever,  a  task  can  be  practiced  on  obsolete  equipment.  HumRRO  Work  Unit 
FORECAST  showed  how  a  judicious  combination  of  medium  fidelity  mock-ups 
representing  new  equipment,  plus  the  use  of  obsolete  equipment  to  practice  tasks 
common  to  both  old  and  new  equipment,  caul  reduce  the  amount  of  practice 
required  on  new  equipment  (Shriver,  Fink,  and  Trexler,  37). 
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Whole  Task  va  Task  Component  Trainers 

1 i 

Trainers  for  a  component  of  a  task  can  often  be  less  expensive  than  a  trainer 
designed  to  permit  practice  on  the  entire  task.  V  'hether  to  use  part  task  trainers 
depends  on  whether  transfer  from  the  component  to  the  task  will  be  expected. 
Earlier  in  this  chapter  it  was  pointed  out  that  such  transfer  can  generally  be 
expected,  unless  there  is  time- sharing  or  interaction  between  the  components. 

THE  ROLE  OF  THE  INSTRUCTOR 

The  human  instructor  can  perform  several  functions  with  regard  to  the 
practice  of  performance.  He  may  instruct  the  student  as  to  how  to  practice;  he 
may  monitor  the  class  to  determine  whether  instructions  are  being  followed; 
and  he  may  provide  KOR. 

As  a  giver  of  instructions  he  may  be  replaced— or  his  work  supplemented— by 
written  materials,  tape  recordings,  films,  and  television.  The  choice  among 
these  would  depend  on  the  relative  cost  of  each,  since  they  are  generally 
equally  effective. 

As  a  monitor,  he  can  determine  fairly  easily  whether  students  are  working 
at  the  task  being  practiced,  provided  he  is  not  required  to  supervise  too  many 
students  at  one  time.  When  the  instructor  is  performing  as  a  monitor  he  is  a 
part  of  the  management  of  students  function. 

If  the  instructor  must  provide  KOR  in  addition  to  monitoring,  even  a  few 
students  will  constitute  a  serious  overload.  It  will  be  impossible  for  him  to 
observe  each  student  as  he  practices  and,  at  the  same  time,  provide  KOR  to 
each  student  at  the  proper  time. 
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Chapter  3 

THE  PRACTICE  OF  KNOWLEDGE 


INTRODUCTION 
What  is  Knowledge  ? 

As  used  in  this  series  of  reports,  knowledge  refers  to  symbolic  processes. 
The  symbols  include  words,  numbers,  mathematical  formulas,  codes,  and  pic¬ 
tures.  In  distinction  to  the  previous  chapter,  which  concerns  the  practice  of 
the  task  itself,  this  chapter  deals  with  the  practice  of  the  knowledge  components 
of  the  task. 

These  knowledge  components  typically  include  such  items  as  the  nomen¬ 
clature,  location,  and  function  of  equipment;  descriptions  of  step-by-step  pro¬ 
cedures;  special  codes,  such  as  map  symbols,  schematic  diagrams,  and 
blueprint  symbols;  the  appearance  of  various  items  in  the  work  environment; 
principles  that  apply  to  many  tasks;  and  precautions  to  prevent  injury  to  per¬ 
sonnel  or  damage  to  equipment. 

Purpose  of  Practicing  Knowledge 

Practice  of  knowledge  components  will  transfer  to  the  learning  of  the  task 
provided  the  knowledge  is  thoroughly  learned.  This  transfer  will  reduce  the 
requirement  for  time  to  practice  the  task,  since  the  task  will  be  learned  sooner. 
In  turn,  this  may  be  reflected  in  a  smaller  requirement  for  training  equipment.' 

The  kinds  of  knowledge  that  will  transfer  to  the  practice  and  performance 
of  a  task  will  be  those  identified  as  a  component  of  that  task  in  the  hierarchy 
of  objectives.  The  process  of  determining  knowledge  and  skill  components  of  a 
task  was  described  in  the  earlier  training  objectives  report  <1),  and  will  not  be 
repeated  here. 

The  evidence  is  also  clear  on  another  point:  Little  transfer  can  be  expected 
unless  the  student  has  attained  a  high  degree  of  meatery  of  the  knowledge  com¬ 
ponent.  Mastery  depends  on  sufficient  practice;  one  presentation  of  the  infor¬ 
mation  is  seldom  enough. 

A  third  factor  to  consider  is  that  it  is  often  much  less  expensive  to  practice 
knowledge  than  to  practice  performance.  To  practice  a  task  requires  tools  and 
equipment;  in  most  situations,  to  practice  knowledge  requires  little  more  than 
paper  and  pencil. 

Relation  of  Practice  to  Presentation 

Clearly,  before  knowledge  can  be  practiced,  it  must  be  presented  to  the 
student;  however,  the  methods  of  presentation  may  differ.  For  example,  it  may 

‘The  research  literature  amply  supports  the  proposition  that  learning  the  proper  kind  of  knowledge 
transfers  positively  to  practice  of  performance.  Among  the  pertinent  studies  are  those  of  Arnoult  (38,39); 
Bailey  and  Jeffrey  (40);  Baker  and  Wylie  (41);  de  Rivera  (42);  Eckslrand  and  Morgan  (43);  Gagnd  and 
Baker  (44);  Gagni,  Baker,  and  Foster  (45);  Hake  and  Eriksen  (48);  Kinkade  and  Kidd  (47);  McAllister  (48); 
Rossman  and  Goss  (49);  Holton  and  Goss  (SO);  Goes  and  Greenfeld  (51). 
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be  presented  in  small  or  large  segments,  before  a  student  practices  or  con¬ 
currently  with  practice. 

For  this  report,  presentation  and  practice  of  knowledge  are  differentiated 
on  the  basis  of  a  one-way  versus  two-way  relationship  between  the  student  and 
the  media  used  in  presenting  and  practicing  knowledge.  In  presentation,  -’here 
is  a  one-way  flow  from  the  media  to  the  student.  In  practice,  a  knowledge  cue 
(usually  a  small  amount  at  a  time)  is  presented;  the  student  responds,  f.nd 
then  is  provided  knowledge  of  results. 

In  dealing  with  practice,  then,  it  will  be  assumed  that  small  amounts  of 
knowledge  will  be  presented  during  practice.  The  function  of  presentation 
described  in  the  next  chapter,  will  include  only  the  one-way  flow  of  know,  .ge 
from  an  appropriate  medium  to  the  Student. 

ANALYSIS  OF  KNOWLEDGE  ^ 

Before  materials  and  techniques  for  the  practice  of  knowledge  are  .^signed, 
the  knowledge  component  should  be  analyzed  to  determine  the  proper  cues, 
responses,  and  KOR  so  that  the  student  may  learn  efficiently  (Gilbert,  S2; 
Hoehn,  53).  The  type  of  analysis  to  be  conducted  will  depend  on  whether  thv 
knowledge  falls  into  one  of  several  broad  classes,  described  below. 

In  the  design  of  practice  materials,  results  of  the  analysis  of  the  knowledge 
component  must  be  considered.  The  cues  (or  a  symbolic  representation  of  them) 
must  be  presented.  Opportunity  for  a  symbolic  response  must  be  provided. 
KOR  must  also  be  provided  to  the  student. 

Discrimination 

If  an  individual  is  presented  with  a  series  of  cues,  and  each  cue  requires 
a  different  response,  the  behavior  required  is  a  form  of  discrimination.  This 
situation  may  be  diagrammed  in  the  following  manner; 

Cue  1  — .. — «.  Response  1 

Cue  2  »  Response  2 

Cue  3  . .  -■♦  Response  3 

etc. 

An  essential  point  to  emphasize  with  regard  to  discriminations  is  that  they 
are  directional.  That  is,  the  task  requires  certain  responses  to  specific  cues. 
Instructors  should  not  require  their  studentB  to  practice  such  discriminations 
in  reverse  order. 

It  should  be  noted  that  in  some  instances  the  student  does  need  to  learn 
discriminations  both  ways.  That  is,  the  cue  in  one  situation  will  be  the  response 
in  another,  and  vice  versa. 

What  kinds  of  knowledge  will  be  in  the  form  of  discriminations  ?  Common 
examples  are: 

(1)  Nomenclature  of  work  objects.  Here  the  picture  or  drawing  of  the 
object  will  be  the  cue,  and  the  name  the  response. 

(2)  Functions  of  equipment.  The  cue  is  the  name  or  picture,  while 
the  response  is  a  statement  of  the  function  of  the  equipment. 

(3)  Locations  of  equipment  components.  Here  the  cue  is  a  name  or 
a  picture,  and  the  response  will  be  the  location,  either  in  words  or  pointing  it 
out  in  a  drawing. 

(4)  Symbols.  Symbols  occur  in  many  tasks.  Maps,  blueprints,  elec¬ 
tronic  schematic  drawings,  all  use  symbols  that  must  be  learned.  Here  the  cue 
will  be  the  symbol,  and  the  response  either  the  name  or  recognition  of  a  picture 
of  what  the  symbol  stands  for. 


22 


In  discrimination  learning,  sufficient  practice  should  be  provided  to  elim- 
i,  :ai  ."-rorri  of  discrimination- that  is,  the  giving  of  an  incorrect  response 
It  will  be  helpful  if  the  differences  between  cues  are  identified  and 
■ .  out  to  the  student. 


Com  i  at  Formation  or  Generalization 

\  uch  of  the  learning  of  knowledge  involves  the  learning  of  concepts.  A 
"i  ■  <  pt  is  an  abstraction  or  a  term,  which  may  apply  to  many  specific  instances 

e-r  •  samples-. 

There  are  two  kinds  of  knowledge  related  to  a  concept  that  should  be 
1<  ied.  Fi "ct,  there  are  the  various  defining  characteristics  of  the  concept— 
it  definition.  Second,  there  are  a  number  of  specific  instances  of  the  concept. 

For  example,  the  concept  and  its  definition  can  be  diagrammed  as  follows: 

Concept - !— ►  Definition 

(Cue)  (Response) 

Of  course,  if  required  by  the  task,  the  concept  may  be  diagrammed  thus: 

Specific  Instances 
(Response  1) 

Concenter — '  —  (Response  2) 

(Cue)  (Response  3) 

(Response  4) 
or). 

Specific  Instances 
(Cue 
(Cue 
(Cue 
(Cue 

depending  on  the  task  requirements. 

In  learning  concepts,  the  student  must  generalize  particular  aspects  of  the 
specific  instances  to  form  a  concept,  or  must  identify  what  aspects  of  the  spe¬ 
cific  instances  make  them  belong  to  a  general  class. 

Another  aspect  of  the  learning  of  concepts  is  discrimination.  Concepts  may 
have  to  be  discriminated  from  each  other,  and  the  general  characteristic  of  the 
specific  instances  related  to  the  concept  may  have  to  be  discriminated  from 
characteristics  not  related  to  the  concept. 

Serial  Learning 

.  Ji 

In  the  earlier  report  oii  objectives  (f),  it  was  pointed  out  that  many  tnsks 
take  the  form  of  fixed  or  variable  procedures.  A  form  of  knowledge  component 
appropriate  to  procedures  is  thev  sequence  of  step-by-step  cues,  actions,  and 
indications  of  correct  action  involved  in  the  task. 

Fixed  procedures  have  no  deviations  or  choice  points  in  them.  Variable 
procedures  have  choice  points,'  in  which,  depending  on  the  appropriate  cue, 
different  chains  of  activity  are  chosen.  For  variable  procedures,  it  is  impor¬ 
tant  to  identify  the  cues  that  require  making  various  choices.  Cues  requiring 
a  choice  of  actions  to  be  taken  may  be  analyzed  as  a  discrimination  problem 
for  the  student. 

Fixed  procedures  and  the  chain  portions  of  variable  procedures  are  repre¬ 
sented  as  follows: 

Cue  —  Action-— Indication,  »  Cuey— »•  Action  — —  Indication  etc. 


Concept 

(Response) 
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The  detailed  task  description  for  the  procedure  will  show  whether  the 
particular  procedure  is  fixed  or  variable  in  form. 

Principles 

A  principle  is  a  more  or  less  general  statement  of  relationships,  usually 
involving  two  or  more  concepts.  The  concepts  included  in  a  principle  should  be 
analyzed  in  accordance  with  the  techniques  dealing  with  cues  and  responses, 
described  previously. 

As  Miller  (32)  has  pointed  out,  unless  the  principle  is  anchored  in  the  cues 
and  actions  required  by  the  job,  it  will  remain  "ivory  tower  knowledge,"  of  little 
use  on  the  job.  It  is,  then,  necessary  to  describe  the  cues  in  the  environment 
that  will  call  for  application  of  the  principle,  and  the  specific  actions  called  for 
by  the  principle.  Then,  when  the  principle  is  taught,  the  student  should  have  a 
better  chance  of  applying  it. 

Principles  should  be  analyzed  in  the  following  manner: 

Cues  — —  (Principle) - -  Actions 

Principles  of  broad  generality  will  have  many  specific  cues  and  many  spe¬ 
cific  actions.  Care  should  be  taken  to  identify  a  reasonable  sample  of  actual 
cues  and  actions  which  might  .be  taken. 

FACTORS  AFFECTING  PRACTICE  OF  KNOWLEDGE 
Knowledge  of  Results 

KOR  is  as  important  in  the  learning  of  knowledge  as  in  the  learning  of  per¬ 
formance.  KOR  must  be  provided  to  the  student  when  he  practices.' 

Two  studies  (Hirsch,  54;  Krumboltz  and  Bonawitz,  55)  have  shown  that  it 
is  generally  desirable,  in  providing  KOR  during  the  learning  of  knowledge,  to 
make  the  KOR  meaningful.  That  is,  instead  of  simply  telling  the  student  "right" 
or  "wrong, "  ot  simply  giving  the  correct  answer,  it  is  desirable  to  give  an  expla 
nation  for  the  correct  answer,  if  practical. 

Bryan,  Rigney,  and  Van  Horn  (56)  compared  the  effectiveness  of  several 
meaningful  forms  of  KOR,  and  found  them  equally  effective.  They  employed  the 
following  forms  of  explanations: 

(1)  Those  that  gave  the  trainee  the  correct  definition  or  description  of 
the  chosen  alternative, 

(2)  Those  that  provided  the  principal  reason  why  the  chosen  alterna¬ 
tive  was  keyed  as  correct  or  incorrect. 

(3)  Those  that  pointed  out  the  probable  operational  consequences  of 
the  course  of  action  represented  by  the  alternative. 

The  course  designer  should  not  be  surprised  to  discover  that  his  students 
do  not  make  use  of  all  the  KOR  he  provides.  The  student  uses  any  form  of  KOR 
he  feels  is  helpful  to  him.  If  he  is  confident  he  knows  the  answer,  he  may  pay 
little  attention  to  the  KOR  provided  him.  This  does  not  mean  that  KOR  should 
not  be  made  available.  Students  "fill  differ  in  their  need  for  KOR,  and  the  stu¬ 
dent's  confidence,  of  course,  may  be  misplaced. 

MEANINGFULNESS 

The  meaningfulness  of  knowledge  is  a  very  important  variable  in  regard  to 
both  learning  and  retention  of  knowledge  (Dowling  and  Braun,  57;  Noble,  58; 
Sarason,  59).  The  more  meaningful  the  material  is  to  the  student,  the  easier 


'See  the  diacuaiion  of  KOR  in  Chapter  2. 
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it  is  to  learn  and  recall.  A  foundation  of  meaningfulness  developed  through 
effective  sequencing  of  objectives,  promoted  through  the  presentation  of  knowl¬ 
edge,  and  carried  through  the  practice  of  knowledges  and  skills  will  be  a  power¬ 
ful  aid  to  learning. 

The  principle  of  making  knowledge  meaningful  can  be  applied  in  many  ways 
in  the  practice  of  knowledge.  Other  things  being  equal,  knowledge  that  can  be 
related  to  student  experience  will  be  more  meaningful  than  that  which  cannot. 
Knowledge  that  can  be  clearly  related  to  the  learning  of  a  job  task  will  be  more 
meaningful  than  that  which  cannot.  Knowledge  that  is  organized  and  classified 
will  be  more  meaningful  than  that  which  is  not.  Knowledge  related  to  signifi¬ 
cant  student  goals  will  be  more  meaningful  than  that  which  is  not.. 

One  practical  application  of  meaningfulness  in  the  practice  of  knowledge 
can  be  found  in  the  research  data  dealing  with  the  classification  and  organiza¬ 
tion  of  knowledge  to  be  learned.  Hake  and  Eriksen  (60),  Helson  and  Cover  (61), 
and  Mathews  (6 2)  have  shown  that  classifying  knowledge  into  categories  helps 
both  initial  learning  and  retention.  Another  suggestion,  made  by  Hoehn  (53),  is 
to  avoid  lengthy  drill  sessions  on  such  relatively  meaningless  material  as  the 
nomenclature  and  the  location  of  equipment  or  its  parts.  Instead,  combine  this 
practice  with  the  learning  of  the  function  of  the  equipment  or  its  parts.  Still 
another  suggestion  is  to  teach  the  hierarchical  structure  of  the  ''bjectives  so 
that  the  student  can  see  the  relationship  between  practice  of  a  particular  item 
of  knowledge  and  the  learning  of  a  task. 

Overt  and  Covert  Response 

Let  us  imagine  that  we  have  designed  materials  for  practicing  knowledge 
so  that  the  student  must  respond  when  presented  with  a  cue.  Should  we  require 
the  student  to  make  an  overt  response,  such  as  writing  his  answer,  or  is  it 
sufficient  to  ask  him  just  to  think  his  answer  ? 

The  research’  is  very  consistent  on  one  point:  It  takes  less  time  for  the 
student  to  think  his  answer  than  to  write  it  (Goldbeck  and  Campbell,  63; 
Krumboltz  and  Weisman,  64;  Lambert,  Miller,  and  Wiley,  65;  Stolurow  and 
Walker,  66).  These  studies  also  generally  show  that  a  covert  response  mode 
is  as  effective  as  an  overt  mode  for  initial  learning;  however,  Krumboltz  and 
Weisman  found  that  retention  was  superior  with  ah  overt  response  mode. 

Discovery  of  Principles 

In  teaching  a  principle,  is  it  better  simply  to  present  the  principle  to  the 
student,  along  with  appropriate  cues  and  actions,  or  is  it  better  to  try  to  get 
the  student  to  discover  the  principle  himself?  Several  studies  have  shown  that 
arranging  for  the  student  to  discover  the  principle  leads  to  higher  levels  of 
learning  (DiVesta  and  Blake,  *7;  Forgus  and  Schwartz,  68;  Kersh,  69)  or  higher 
levels  of  retention  and  transfer  (Haslerud  and  Meyers,  70). 

One  way  to  apply  this  finding  is  to  ask  the  student  to  look  for  and  find  a 
principle  in  the  material  he  has  been  given.  He  can  be  given  various  kinds  of 
prompts  and  hints.  There  is  a  possibility,  however,  that  the  use  of  discovery 
maybe  more  time-consuming  than  simply  presenting  the  principle  to  the  student. 

Distribution  of  Practice 

Whether  practice  situations  are  distributed  throughout  training  or  grouped 
with  little  interval  between  practice  periods  is  less  significant  in  learning 

’Most  of  the  recent  research  on  this  point  hss  been  associated  with  programed  instruction. 
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knowledge  than  in  learning  performance  (Underwood,  71).  In  the  practice  of 
knowledge,  the  only  situation  in  which  distribution  of  practice  has  much  effect 
is  when  there  io  interference  between  the  responses.  If  a  student  tends  to  give 
erroneous  responses  to  specific  cues,  the  interval  between  trials  should  be 
kept  short,  on  the  order  of  a  few  seconds. 

Other  Factors 

Some  additional  factors  have  been  found  to  influence  the  learning  Of  knowl¬ 
edge  in  specific  circumstances. 

Arnoult  (72)  taught  subjects  to  recognize  spatial  patterns  by  one  of  several 
methods:  the  reproduction  Of  patterns  from  memory,  general  questioning  about 
patterns,  specific  questioning  about  patterns,  and  observation  of  patterns.  The 
reproduction  method  was  clearly  superior  to  the  others,  while  the  questioning 
methods  were  relatively  ineffective. 

Notetaking  during  lectures  may  be  considered  a  rough  form  of  practice.  In 
one  study  that  involved  comparison  of  two  groups  (Eisner  and  Rohde,  73V,  one 
group  took  notes  during  the  lecture  and  studied  later,  while  the  other  took  notes 
and  studied  immediately  after  the  lecture.  There  was  no  difference  in  achieve¬ 
ment  between  the  groups.  McClendon  (74)  compared  the  following  procedures; 
taking  no  notes  on  a  lecture,  taking  notes  on  the  main  points  only,  taking  detailed 
notes,  and  taking  notes  in  the  student's  usual  manner.  There  were  no  differences 
in  achievement  or  retention  because  of  different  notetaking  practices. 

TECHNIQUES  FOR  PRACTICING  KNOWLEDGE 
Basic  Patterns  for  Practicing  Knowledge 

The  most  usual  pattern  for  the  practice  of  knowledge  is  to  present  the  cue 
(or  a  symbolic  representation,  such  as  a  drawing,  diagram,  photogrr.ph,  or 
word(s)),  and  require  the  student  to  produce  (or  think)  the  response,  followed 
by  KOR.  There  are  special  considerations  that  apply  to  specific  kinds  of  prac¬ 
tice  (Hoehn,  53). 

In  discrimination  learning,  it  is  important  that  the  attention  of  the  trainee  be 
directed  to  the  critical" aspects  of  each  cue  that  differentiates  it  from  other  cues. 

In  the  learning  of  concepts,  the  trainee  should  be  presented  with  cues  in 
which  the  common  characteristic  that  defines  the  concept  appears,  but  in  which 
other  features  vary  as  they  would  be  expected  to  in  the  job  environment.  The 
trainee  should  receive  practice  in  which  he  discriminates  between  cues  that 
contain  the  common  characteristic  and  those  that  do  not. 

In  the  learning  of  procedures,  an  effective  practice  situation  can  be  devel¬ 
oped  from  drawings,  or  photographs  of  equipment  panels  and  consoles.  This 
will  enable  the  stiident  to  practice  knowledge  not  only  of  the  location  of  controls 
and  indicators  but  of  the  sequence  of  actions. 

Lengthy  procedures  can  often  be  subdivided  into  portions  ’bat  have  a  com¬ 
mon  name  and  purpose.  Teaching  the  student  this  information  along  with  the 
procedure  will  help  him  organize  the  learning  and  will  make  it  more  meaningful. 

In  the  teaching  of  principles  the  possibility  of  learning  by  discovery  has 
already  been  mentioned.  Care  must  be  exercised  to  provide  the  student  with  a 
reasonable  sample  of  cues  and  actions  to  which  the  principle  is  related. 

Self- Instructional  Techniques 

Wittyp  the  field  hf  programed  learning,  a  variety  of  techniques  have  been 
developed  to  permit  the  student  to  practice  knowledge  by  himself  and  at  bis  own 
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rate.’  These  include  linear  :rograms,  branching  programs,  and  various  com¬ 
binations  of  each.  The  self-instructional  methods  permit  the  instructor  to 
monitor  progress  of  the  class  as  a  whole  and  to  work  with  individual  students. 

Various  types  of  worksheets  and  workbooks  can  be  developed  to  permit 
student  practice  of  knowledge  within  the  guidelines  of  this  chapter.  These  can 
be  organized  to  present  cues,  permit  the  student  to  respond,  and  provide  knowl¬ 
edge  of  results.  They  may  be  supplemented  by  a  variety  of  response  devices, 
such  as  punchboards  and  special  answer  sheets  (Pressey,  84;  Schutz  and 
Whitt emore,  85). 

Development  of  specific  methods  for  providing  practice  within  a  self- 
instructional  framework,  using  the  guidelines  of  this  chapter,  is  limited  only 
by  the  ingenuity  of  the  instructional  system  designer. 

Classroom  Feedback  Methods 

Practice  of  knowledge  can  also  be  provided  through  classroom  feedback. 
These  techniques  invo’ve  more  instructor  activity  and  control  of  the  practice 
session  than  do  the  individualized  methods.  They  allow  students  to  practice  at 
a  pace  set  by  the  instructor,  and  permit  a  considerable  degree  of  control  by  the 
instructor.  In  general,  they  require  the  instructor  to  present  questions  and 
problems  to  the  class,  followed  by  an  answer  from  each  student,  followed  by 
KOR  from  the  instructor  (feedback). 

Several  electrical  devices  have  been  built  to  provide  classroom  feedback. 
Usually,  after  the  instructor  asks  a  question,  or  presents  a  slide  with  a  ques¬ 
tion  on  it,  the  student  presses  a  button  or  other  control  indicating  one  of  several 
responses.  A  control  panel  shows  the  instructor  how  many  students  gave  the 
correct  response.  He  can  then  provide  KOR. 

A  simpler  version  of  the  same  technique  is  to  give  each  student  four  or  five 
cards,  each  a  different  color.  When  the  instructor  asks  a  question,  the  student 
holds  up  the  card  that  represents  the  alternative  he  has  selected.  The  instructor 
then  provides  KOR. 

Student -Coach  Method 

The  student-coach  method  permits  considerable  student  activity  under  gen¬ 
eral  rronitdrship  of  the  instructor.  In  the  student -coach  method,  one  student 
i3  the  learner,  while  another  stiSdent  asks  questions  and  gives  KOR.  Naturally, 
the  coach  must  have  the  appropriate  materials  to  use. 

It  is  suggested  that  the  coac*  and  student  alternate  roles  in  successive 
trials,  so  that  each  trainee  can  be  conch  and  student.  It  is  very  likely  that 
the  trainees  will  learn  during  both  roles . 

Interest  may  be  added  by  keeping  score?  on  the  number  of  questions  answered 
correctly  by  the  student  on  each  trial.  Each  3tudent  can  informally  compete 
both  with  himself  and  with  the  other  member  of  the  pair. 

The  Instructor 

All  of  the  methods  described  in  the  preceding  pages  are  intended  to  elimi¬ 
nate  the  problem  of  overloading  the  instructor  during  the  practice  session.  The 
instructor  cannot  observe  all  that  is  going  on  in  any  degree  of  detail,  and  pro¬ 
vide  appropriate  KOR,  if  he  has  more  than  two  or  three  students. 

'The  reader  ia  referred  to  studies  by  Gilbert  (52);  Green  (7$);  Lumsdaine  and  Gianer  (76);  by  naught  (77); 
Melching,  et  al.  (78);  Ofiesh  and  Meierhenry  (79);  Schramm  (80);  Stolurow  (81);  Melching,  et  at.  (82); 

DA  PAM  690-21  (83). 
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Chapter  4 

THE  PRESENTATION  OF  KNOWLEDGE 


THE  ROLE  OF  PRESENTATION 
A  Contrast  With  Practice 

During  the  practice  of  both  performance  and  knowledge,  the  student  responds 
actively,  and  receives  KOR.  In  the  presentation  of  knowledge,  he  is  relatively 
passive.  He  receives  the  presentation;  it  is  to  be  hoped  that  he  pays  attention; 
but  he  does  not  respond  and  does  not  receive  KOR. 

It  is  easier  to  design  situations  for  presentation  than  for  practice.  The 
lack  of  interaction  between  the  student  and  his  environment  poses  limitations  on 
the  effectiveness  of  presentation  as  a  means  of  teaching;  however,  presentation 
can  contribute  significantly  to  the  development  of  performance. 

Purposes  Served  by  Presentation 

Providing  Orientation.  It  is  generally  useful  to  provide  the  student  with  an 
overview  of  the  job  he  is  training  for.  This  permits  him  to  relate  specific  train¬ 
ing  activities  to  the  overall  purpose  of  his  training.  Provision  of  this  orienta¬ 
tion  is  useful  even  though  no  specific  training  objective  may  be  served  by  it. 

Providing  Organization  and  Meaning  to  Knowledge  Items.  Meaningfulness, 
which  was  discussed  in  connection  with  practice  of  knowledge,  is  also  an  impor¬ 
tant  factor  for  the  presentation  of  knowledge,  particularly  because  of  the  stu¬ 
dent's  passive  role  in  presentation.  Presentation  techniques  can  be  designed  to 
provide  the  student  with  a  suitable  framework  of  organization  and  meaning  that 
will  help  him  learn  specific  knowledge  content. 

Guiding  Practice.  In  the  initial  stages  of  practice  of  either  performance  or 
knowledge,  presentations  can  guide  the  student’s  practice  so  that  he  makes 
fewer  errors.  For  example,  picture  guides  including  both  step-by-step  proce¬ 
dures  and  appropriate  photographs  of  the  equipment  were  developed  in  HumRRO 
SHOCKACTION  research  (MacCaslin,  Woodruff,  and  Baker,  86).  These  guides 
helped  the  students  practice  effectively.  A  variety  of  devices,  which  include 
both  slide  and  sound  presentations,  are  available  to  guide  practice.  Demonstra¬ 
tions,  of  coursew  are  widely  used  for  this  purpose. 

Presenting  Knowledge  to  be  Learned.  In  most  training  courses,  a  great 
deal  of  time  is  spent  presenting  knowledge  to  be  learned,  using  lectures,  demon¬ 
strations,  films,  and  so  forth.  Although  students  do  learn  from  presentation 
alone,  there  are  limitations  on  the  amount  that  can  be  so  learned.  Learning  is 
more  effective  when  the  student  is  actively  responding  and  receiving  KOR  than 
when  he  is  passively  receiving  information, 

A  related  problem  in  presentation  without  student  response  is  the  dif¬ 
ficulty  in  determining  immediately  whether  the  student  understands  the  message, 
whether  it  is  being  delivered  too  fast  or  too  slowly,  or  whether  the  student  was 
paying  attention. 


26 


i 


Methods  and  Media 

Among  the  wide  variety  of  media  and  methods  developed  for  the  presentation 
of  knowledge  are  lectures,  demonstrations,  films,  television,  training  aids, 
slides  and  tape  recorders,  and  books.  Previous  research  evidence  suggest'*? 
that,  although  each  has  special  capabilities,  these  methods  and  media  are 
equally  effective;  therefore,  a  major  factor  in  the  selection  of  methods  and  media 
is  relative  cost.  Cost  consideration  should  include  not  only  initial  cost,  but  also 
replacement  and  repair  of  equipment, 

GENERAL  GUIDELINES  FOR  DESIGN  OF  PRESENTATIONS 
Meaning  and  Organization 

It  is  important  that  meaningfulness  and  organization  be  built  into  the  pres¬ 
entation.  Meaning  can  be  designed  into  it  by: 

(1)  Providing  overviews  of  the  job. 

(2)  Relating  specific  knowledge  to  task  performance. 

(3)  Relating  specific  knowledge  to  trainee  experience,  interests, 
and  goals. 

Much  of  the  material  presented  will  have  a  natural  organization.  Where 
this  is  the  case,  the  natural  organization  should  be  followed.  Otherwise  the 
material  should  be  organized  or  classified  by  the  instructional  system  designer. 
Organization  of  material  can  be  achieved  by  the  use  of: 

(1)  Outlines  showing  the  structure  of  the  knowledge  being  presented. 

(2)  Topic  headings  showing  the  main  points  being  presented. 

(3)  Charts  and  graphs  showing  the  organization  of  the  hierarchy 
of  objectives. 

(4)  Repetition  during  the  presentation  of  the  main  points  or  major 
outline  headings. 

A  Set  to  Learn 

It  is  necessary  to  create  in  the  student  a  set  to  learn,  that  is,  a  motivation 
to  pay  attention  to,  and  to  learn,  the  material  needed  for  job  performance. 
There  is  much  evidence  that  students  will  generally  learn  what  they  think  they 
should,  which  is  not  necessarily  what  they  were  supposed  to  learn.  What  is 
needed,  then,  is  to  induce  in  the  student  a  set  to  learn,  and  also  help  him  identify 
the  material  that  is  most  important. 

Among  the  several  methods  that  can  be  used  to  create  this  set  are 
the  following: 

(1)  Providing  the  student  with  a  list  of  the  objectives. 

(2)  Administering  a  short  quiz  or  test  before  the  presentation. 

(3)  Notifying  students  that  they  will  be  tested  on  the  material  presented. 

(4)  Giving  •  preview  that  points  out  the  main  topics  to  be  presented. 

Assuring  Audience  Rec.  >tion 

Whereas  in  the  practice  of  performance  and  knowledge  the  student  makes 
responses  that  can  be  corrected  if  they  are  wrong,  during  a  presentation  the 
student  i3  simply  watching  or  listening.  The  human  lecturer  or  demonstrator 
or  the  presentation  device  does  not  know  whether  the  audience  is  receiving  the 
message.  Material  may  be  presented  too  fast,  too  much  new  material  may  be 
presented  at  one  time,  or  the  language  and  pictures  may  be  confusing. 
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Since  there  is  no  feedback  from  the  student  during  the  presentation,  the 
presentation  should  be  tested  to  determine  whether  the  message  is  clear  and 
appropriately  paced  for  the  student  audience.  The  general  pattern  of  such  a 
test  would  be  as  follows: 

(1)  Present  a  short  section  of  the  message. 

(2)  Ask  typical  students  questions  to  determine  whether  they  received 
the  part  of  the  message. 

(3)  Repeat  steps  1  and  2  until  the  total  message  has  been  tested. 

(4)  Analyze  the  answers  to  these  questions. 

(5)  Revise  the  presentation,  if  necessary. 

(6)  Retest  or  revise  the  presentation  until  it  clearly  communicates  to 
the  majority  of  the  students. 


TRAINING  FILMS 


Extensive  research  has  been  conducted  on  the  use  of  films  for  training,  and 
certain  research  findings  are  of  interest  to  the  overall  planner  of  the  instruc¬ 
tional  system.  The  most  comprehensive  series  of  studies  is  that  conducted  by 
the  Instructional  Film  Research  Program  of  Pennsylvania  State  University 
(Carpenter,  87;  U.S.  Naval  Training  Device  Center,  88).  The  following  material 
is  based  primarily  on  these  studies.. 

Implications  for  the  Instructional  System 

Films  can  present  knowledge  with  equal  or  greater  effectiveness  than  other 
comparable  methods  and  have  a  wide  variety  of  applications.  They  have  been 
used  to  teach  factual  information,  foreign-language  vocabulary,  motor  skills, 
and  mental  hygiene,  and  to  instill  attitudes. 

Training  films  are  most  effective  when  they  are  designed  to  accomplish 
specific  training  objectives  with  a specific Audience.  For  example,  films  with 
general  and  rather  superficial  content  aimed  at  a  wide  audience  are  likely  to 
be  less  effective  than  films  with  specific  content,  related  to  objectives,  and 
aimed  at  an  audience  with  known  characteristics. 

Students  can  learn  how  to  learn  from  films.  A  later  section  on  suggestions 
for  film  use  will  describe  ways  by  which  the  student  can  be  taught  to  learn  more 
effectively  from  films. 

Training  films  can  be  produced  inexpensively  without  sacrificing  effective¬ 
ness.  Many  of  the  techniques  used  in  films  designed  for  entertainment  need 
not  be  used  because  they  do  not  contribute  to  training.  Among  these  are  dra¬ 
matic  sequences,  special  camera  effects,  and  color. 

Implications  for  Film  Production 

Research  has  suggested  a  number  of  ways  to  increase  the  effectiveness  and 
reduce  the  cost  of  training  films. 

First*  sound  principles  of  learning  must  be  applied  in  the  film.  A  number  of 
these  points  will  be  noted  as  similar  to  those  described  in  other  contexts;  some 
will  also  suggest  wajrs  in  which  practice  of  either  performance  or  knowledge  can 
be  integrated  with  film  presentations. 

The  need  to  consider  the  audience  has  already  been  mentioned.  More  spe¬ 
cific  suggestions  are: 

(1)  Show  performance  on  the  screen  as  if  the  learner  were  doing  the 
job  himself.  Show  him  what  he  would  see. 
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(2)  Present  information  either  by  picture  or  sound  at  a  rate  slow 
enough  to  permit  the  student  to  receive  the  message.  Do  not  introduce  too 
many  new  ideas.  Avoid  packing  the  sound  track  with  too  many  words  per  minute^ 

(3)  Introducing  unnecessary  new  names  or  technical  terms  in  a  film 
places  an  additional  burden  on  the  student  and  may  interfere  with  the  learning 
of  a  performance. 

(4)  Avoid  too  succinct  a  treatment  of  a  topic.  Presenting  only  the 
barest  essentials  or  a  very  rapid  coverage  may  be  completely  ineffective, 

A  number  of  suggestions  can  be  made  to  apply  learning  principles 
already  described: 

(1)  Use  repetition.  Plan  the  organization  of  the  film  so  that  important 
concepts  or  sequences  are  repeated.  The  repetition  of  entire  films,  or  parts 
within  a  film,  is  a  very  effective  means  for  increasing  learning.  In  addition, 
the  important  points  in  the  film  should  be  summarized  sufficiently  completely 
to  serve  as  a  review  and  repetition. 

(2)  Increase  the  organization  of  separate  facts  by  using  an  outline  in 
both  titles  and  commentary. 

(3)  The  meaningfulness  of  the  presentation  to  the  student  can  be 
inci  cased  by  using  the  second  person  (you)  in  commentaries,  speaking  directly 
to  the  student,  and  avoiding  the  passive  voice. 

The  learning  of  skills  can  be  assisted  materially  by  using  films  to  guide 
practice.  For  example: 

(1)  Short  film  loops,  which  can  be  repeated  as  often  as  necessary, 
seem  to  be  an  effective  way  of  teaching  skills. 

(2)  Learning  performance  will  be  improved  if  the  student  practices 
a  skill  while  it  is  being  shown  on  the  screen,  provided  the  film  develops  slowly 
enough,  or  if  periods  of  time  are  planned  to  permit  the  student  to  practice 
without  missing  new  content. 

Many  of  the  techniques  used  in.  dramatic  or  entertainment  films  do  not  add 
to  the  training  effectiveness  of  films.  Films  that  do  not,  utilize  these  techniques 
can  be  produced  quickly  and  inexpensively.  For  example: 

(1)  Special  effects  designed  as  attention-getting  devices  do  not 
improve  learning. 

(2)  Optical  effects,  such  as  wipes,  fades,  and  dissolves,  do  not  con¬ 
tribute  to  teaching  effectiveness. 

(3)  The  addition  of  stereoscopic  vision  in  one  experiment  did  not 
improve  the  learning  of  a  motor  skill. 

(4)  Research  does  not  show  that  use  of  color  increases  overall  learning. 

(5)  Preliminary  research  suggested  that  music  does  not  increase 
the  effectiveness  of  a  training  film.  „ 

(6)  The  use  of  dramatic  sequences,  comedy,  or  realistic  settings  in 
films  designed  to  teach  factual  material  does  not  increase  learning. 

Suggestions  for  Film  Use 

In  designing  training  situations  in  which  films  are  going  to  be  used, 
adherence  to  certain  practices,  which  are  adaptations  of  familiar  principles, 
has  been  found  to  increase  the  amount  of  learning  from  films. 

First,  it  is  necessary  to  develop  in  the  student  a  set  to  learn  from  films: 

(1)  The  students  should  be  told  firmly  that  they  are  expected  to  learn 
from  the  film.  They  should  be  tested  after  the  film  and  be  informed  in  advance 
that  they  will  be  tested. 
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(2)  Students  should  be  given  printed  study  guides  to  use  before  and 
after  the  film. 

(S)  Students  should  be  discouraged  from  taking  notes,  because  it 
distracts  their  attention. 

(4)  The  purpose  and  importance  of  the  film  should  be  explained. 

The  following  general  suggestions  are  relevant  for  the  effective  use  of  films 
to  promote  learning: 

(1)  Repetitive  showings  increase  learning. 

(2)  Pretesting  and  posttesting  with  provision  for  KOR  add  to  the  effec¬ 
tiveness  of  films. 

(3)  One  showing  of  a  film  describing  a  complex  skill  may  not  be  enough. 
It  is  desirable  to  show  the  film  in  the  area  in  which  students  are  practicing  so 
that  they  can  refer  to  thefilm  as  often  as  they  need  to.  This  can  be  done  by  roar 
projection  daylight  screen.  The  students  should  be  placed  within  12  screen 
widths  of  the  screen,  and  within  30°  of  the  center  line. 

(4)  A  skill  can  be  partly  learned  through  mental  practice.  Students 
should  be  encouraged  to  imagine  they  are  performing  the  skill  while  they  are 
watching  the  film. 

INSTRUCTIONAL  TELEVISION 

Instructional  television,  like  other  media,  must  be  considered  in  the  light  of 
cost,  effectiveness,  r  d  special- capabilities  to  solve  particular  problems. 

The  material  in  this  section  has  been  taken  primarily  from  the  Instructional 
Television  Research  Reports  of  the  U.S.  Naval  Training  Device  Center  (89). 

It  is  strongly  recommended  that  agencies  using  instructional  television  study 
that  report  carefully. 

Criteria  for  Using  Instructional  Television 

The  report  cited  above  describes  13  criteria  by  which  systematic  judgments 
can  be  made  concerning  the  use  of  television  for  presenting  lessons: 

(!)  For  presentation  or  demonstration,  television  is  applicable.  For 
practice,  it  is  not. 

(2)  If  two-way  communication' between  instructor  and  3tudent  is 
necessary,  television  is  not  suitable,  (Both  of  these  criteria  reflect  the  fact 
that  television  is  a  medium  for  presentation.) 

(3)  If  rapid  dissemination  of  information,  reaching  many  men  early  in 
training,  or  rapid  distribution  of  new  information  is  necessary  or  urgent,  tele¬ 
vision  is  favored.  Otherwise,  television  may  be  considered  optional. 

(4)  If  there  is  a  shortage  of  qualified  instructors,  television  is  favo.'sd. 
If  not,  television  is  optional. 

(5)  If  the  training  situation  involves  physical  risk  or  danger,  television 
is  favored.  If  not,  it  is  optional. 

(6)  If  training  aids  or  actual  equipment  is  in  short  supply,  television 
is  favored.  If  supplies  are  ample,  television  is  optional. 

(?)  If  training  aids  and  equipment  are  difficult  to  move  because  they 
are  large,  heavy,  or  unwieldly,  this  circumstance  favors  television.  If  they 
are  easily  moved,  television  is'  optional. 

(8)  If  closeups  are  necessary  in  viewing  training,  this  is  a  plus  factor 
for  television. 

(9)  If  color  is  an  essential  element  of  the  presentation,  color  television 
is  required.  Otherwise,  black  and  white  can  be  used. 
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(10)  If  making  a  sound  record  is  very  desirable,  television  can  be  used 
to  prepare  tapes  or  kinescopes.  If  the  sound  record  is  not  important,  television 
is  optional. 

(11)  If  much  training  time  is  lost  in  moving  from  one  training  ai'ea  to 
another,  television  is  desirable.  Otherwise,  it  is  optional. 

(12)  If  weather  interferes  with  presentation,  use  of  television  can 
solve  this  difficulty. 

(13)  If  material  being  presented  is  classified,  broadcast  television 
cannot  be  used,  although  classified  matter  can  be  transmitted  via  cable. 

The  report  previously  cited  shows  how  these  criteria  can  be  applied  in 
deciding  as  to  the  applicability  of  television  to  particular  lessons. 

The  points  just  set  forth  can  be  summarized  as  follows: 

(1)  Television  is  a  presentation  medium. 

(2)  It  is  an  amplifier  of  presentations.  It  can  give  wide  coverage  to 
both  good  and  poor  presentations.  It  can  amplify  through  closeups  material  that 
would  be  difficult  to  view  directly. 

(3)  It  is  a  form  of  automation,  with  the  advantages  and  limitations 
of  automation. 

(4)  By  tapes  and  kinescopes,  it  provides  a  means  for  making  repeatable 
presentations  quickly. 

Television  and  Films 

Much  of  the  research  conducted  on  the  preparation  and  use  of  training  films 
is  directly  applicable  to  television.  For  instance,  the  use  of  dramatic  special 
effects  does  not  add  to  the  training  effectiveness  of  television.  Television  pres¬ 
entations  can  be  made  more  effective  by  integrating  them  appropriately  with 
practice.  The  reader  interested  in  television  is  referred  to  the  previous  section 
on  films  for  material  on  this  point. 

Technical  Matters 

Television  is  a  medium  with  a  high  initial  cost.  It  requires,  a  group  cf 
trained  operators  and  maintenance  personnel.  There  are  many  technical  matters 
concerning  equipment  and  its  use,  and  production  techniques  that  must  be  under¬ 
stood.  These  are  beyond  the  scope  of  this  paper,  The  U.S.  Naval  Training  Device 
Center  report  (89)  treats,  intensively,  the  production  techniques  appropriate  to 

training.  Additional  sources  are  included  in, the  list  of  references  for  this  chapter. 

•  ,u  * 

Training  vs  Entertainment* 

One  management  problem  concerned  with  the  use  of  television  requires 
special  consideration,  It  is  likely  that  the  key  production  personnel  acquired 
by  a  training  agency  will  have  been  trained  and  experienced  in  television  as  an 
entertainment  medium.  They  will  naturally  want  to  employ  these  entertainment 
techniques  for  training.  However,  research  results  clearly  show  that  these 
methods  do  not  contribute  to  training,  and  they  increase  costs  unnecessarily. 

TRAINING  AIDS 

Training  aids  are  generally  used  for  presenting  knowledge  for  pictorial, 
graphic,  or  organizational  content.  They  can  be  used  effectively  for  providing 
students  a  visual  map  of  system  flow  and  Of  relationships  between  system  com¬ 
ponents  in  complex  equipment  (Aukes  and  Simon,  90),  Aukes  and  Simon  also 


33 


show  the  importance  of  not  presenting  unnecessary  information.  If  a  series  of 
conditions  or  subsystems  is  being  taught,  it  is  better  to  prepare  a  series  of  aids, 
each  presenting  only  that  information  relevant  to  a  specific  condition,  than  to 
have  an  overall  aid  that  presents  all  information  at  all  times. 

Training  aids  are  generally  used  in  conjunction  with  lectures,  and  some 
overlap  between  the  material  presented  by  the  lecture  and  that  provided  by  the 
aid  is  likely. 

In  studies  of  the  relative  effectiveness  of  operating  mock-ups,  nonoperating 
mock-ups,  cutaways,  animated  panels,  charts,  and  symbolic  diagrams, 

Swanson  (91)  and  Swanson  and  Aukes  (92)  found  no  significant  differences. 
Torkelson  (93)  compared  the  use  of  mock-ups,  cutaways  and  transparencies 
and  found  no  difference. 

Vris  (94)  and  Silverman  (95)  found  results  which  suggest  that  three- 
dimensional  and  animated  aids  may  be  superior  to  presentation  of  knowledge 
components  of  skilled  performance,. 

On  the  basis  of  this  evidence,  it  is  reasonable  to  use  the  less  expensive 
aid,  except  where  relatively  skilled  performance  is  to  be  developed. 


TAPE  RECORDINGS 

Since  a  lecture  is  primarily  an  auditory  medium,  why  not  substitute  a  tape 
recording  for  the  lecture  ?  Tape  recordings  have  generally  proved  to  be  as 
effective  as  live  lectures.  For  example,  Newman  and  Highland  (96)  compared 
four methods  of  presenting  a  21-hour  course  in  radio  over  a  four-day  period. 

The  methods  were:  an  instructor  (rated  above  average  in  ability);  tape  record¬ 
ings  and  workbook;  chapters  in  a  notebook;  and  tape  recordings  and  slides.  No 
overall  differences  were  found;  the  mass  media  were  as  effective  as  the  instruc¬ 
tor  for  the  first  two-thirds  of  the  course,  while  the  instructor  was  superior  for 
the  last  third.  In  two  separate  studies  of  college  teaching,  Popham  (97,  98) 
compared  tape  recordings  with  live  lecturers  and  found  no  difference.  Follettie(99), 
using  military  trainees  and  content  from  Army  basic  military  training,  found 
no  difference  between  recorded  and  live  lecturers.  These  studies  indicate  a 
tape  recording  is  an  acceptable  substitute  for  a  live  lecture. 


WRITTEN  MATERIALS 


Assuming  that  the  students  are  reasonably  proficient  readers,  textbooks 
and  other  written  materials  are  appropriate  for  presenting  a  wide  range 
.  of  content. 

t 
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Organization 

The  use  of  techniques  for  imposing  meaningful  organization  on  all  training 
content  has  been  emphasized  at  several  points  in  this  report.  Written  materials 
are  no  exception.  Postman  (100)  shows  that  learning  principles  of  organization 
can  increase  the  amount  of  subject  matter  learned; 

Ausubel  (101)  and  Ausubel  and  Fitzgerald  (102)  have  shown  the  value  of 
what  they  call  the  "advance  organizer.”  This  is  an  introductory  statement  that 
describes  the  general  concepts  to  be  presented  in  more  detail  later  in  tnetraining 

Klare,  Mabry,  and  Gustafson  (103)  compared  the  effect  of  underlining  and 
not  underlining  key  words.  They  did  not  tell  their  subjects  the  purpose  of  the 
underlining.  Able  students  were  helped,  but  less  able  students  were  hindered. 
Their  research  indicates  the  importance  of  familiarizing  students  with  any 
organizational  system  used. 
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It  doe3  not  seem  to  make  much  difference  which  organizational  system  is 
used.  Using  two  passages  from  Air  Force  correspondence  courses.  Christenson 
and  Stordahl  (104)  compared  all  possible  combinations  of  the  following:  outline 
at  the  beginning  of  the  passage,  summary  at  the  beginning  of  the  passage,  sum¬ 
mary  at  the  end  of  the  passage,  underlining  of  main  points,  headings  in  state¬ 
ment  form,  and  headings  in  question  form.  They  found  no  differences  in  the 
effectiveness  of  these  treatments. 

Readability 

The  ease  with  which  written  materials  can  be  read  by  readers  of  differing 
educational  levels  can  be  measured  in  several  ways.  These  are  discussed  in 
considerable  detail  by  Dale  and  Chall  (105).  Klare,  Mabry,  and  Gustafson  (106) 
and  Rubenstein  and  Aborn  (107)  have  shown  that  the  more  readable  the  material, 
the  easier  it  is  to  learn. 

Most  techniques  for  increasing  the  ease  of  reading  can  be  reduced  to  two 
basic  rules:  First,  use  short,  easy  words;  second,  use  short  sentences.  The 
value  of  techniques  for  measuring  readability  is  to  provide  the  writer  with  a 
rough  index  as  to  how  difficult  the  material  is  that  he  is  preparing.  Naturally, 
when  the  writer  is  preparing  technical  materials,  it  is  impossible  to  avoid  tech¬ 
nical  words.  However,  care  should  be  used  to  keep  the  nontechnical  content 
as  simple  as  possible. 

THE  LIVE  INSTRUCTOR 

Traditionally  the  live  instructor  is  an  important  medium  for  presenting 
knowledge;  however,  research  clearly  shows  that  he  hap  no  special  advantage 
for  this  function.  The  principal  advantage  to  using  a  live  instructor  is  that  he 
is  traditional:  Students  are  used  to  him. 

The  size  of  his  class  is  limited  by  the  range  of  the  instructor's  voice.  With 
automated  techniques,  there  is  practically  no  limit  to  the  number  of  students  who 
can  be  reached. 

The  principal  disadvantage  of  live  instructors  is  that  they  are  variable. 
Instructors  differ  in  their  ability  to  speak  clearly  and  distinctly.  They  differ  in 
the  rate  at  which  they  speak.  Some  are  more  concerned  with  impressing  stu¬ 
dents  with  how  much  they  know  than  with  teaching. 

The  same  instructor  is  also  uneven  from  time  to  time.  He  may  become 
bored  with  teaching  the  same  material  Over  and  over  again.  He  may  fall  behind 
schedule  and  attempt  to  catch  up  by  talking  faster. 

Excellent  guidance  is  provided  the  live  instructor  by  Department  of  the 
Army  Field  Manual  21-8,  Techniques  of  Instruction  (108). 

OTHER  FACTORS 

Size  of  Class.  There  seems  to  be  no  particular  limit  to  the  number  of 
students  who  can  receive  a  presentation  at  one  time,  as  long  as  ecch  student 
can  receive  the  message. 

Repetition.  Repetition  of  presentations,  or  of  main  points  in  a  lesson,  is 
desirable.  This  factor  is  mentioned  in  the  section  on  training  films.  Anneser 
and  Beecroft  (109)  and  Beecroft  arid  Anneser  (110)  found  that  repeating  the 
main  points  in  a  lecture  was  effective  in  increasing  student  achievement. 
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Chapter  5 

MANAGEMENT  OF  STUDENTS 
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The  purpose  of  the  student  management  function  is  to  keep  the  student  work¬ 
ing  productively  at  the  task  of  learning.  Topics  discussed  in  this  chapter  include 
student  ability  and  motivation,  adjustment  to  individual  differences  in  aptitude, 
maintenance  of  student  interest  in  learning,  and  control  of  avoidable  absences. 

All  training  programs  have  some  kind  of  student  management  function, 
based  primarily  on  monitoring  by  the  instructor  and  the  threat  of  punishment 
for  absences  or  obvious  and  flagrant  inattention.  It  is  suggested  not  that 
such  monitoring  be  eliminated  but  that  it  be  supplemented  by  other,  more 
positive  techniques. 

THE  STUDENT 

Students  differ  in  many  ways*  some  of  which  are  of  considerable  impor¬ 
tance  to  the  design  of  instructional  systems.  Student  characteristics  are 
discussed  here  io  lay  an  appropriate  foundation  for  designing  the  student  man¬ 
agement  function. 

In  order  to  utilize  the  points  discussed  in  this  section,  certain  information 
about  the  students  must  be  collected.  This  should  include:  ' 

(1)  Range  of  aptitude,  to  let  the  designer  know  how  much  adjustment 
will  need  to  be  made  for  individual  differences. 

{£)  Previous  educational  level  and  range,  which  indicate  both  the  level 
at  which  presentations  and  practice  may  be  aimed  and  the  need  to  adjust  for 
individual  differences. 

(3)  Attitudes  toward  previous  education,  to  indicate  whether  there  is  a 
requirement  to  make  the  material  especially  easy  to  learn. 

(4)  Work,  experience  and  bobbies,  to  provide  information  to  guide  selec¬ 
tion  of  examples  and  illustrations,  and  to  provide  clues  for  the  identification  of 
activities  that  may  serve  as  reinforcers.  (See  section  on  "reinforcement".) 

Aptitudes 

The  Nature  of  Aptitudes 

A  common  complaint  among  trainers  is  that  their  students  are  not 
properly  selected.  Of  course,  everyone  wants  students  with  high  aptitude. 
Properly  motivated,  they  practically  train  themselves.  The  difficulty  with 
raising  aptitude  prerequisites  is  that  bright  people  are  at  a  premium;  there  are 
seldom  enough  of  them  to  go  around. 

An  aptitude  area  score  is  a  weighted  sum  of  the  scores  from  a  large 
group  of  tests.  These  scores  are  developed  on  the  basis  of  many  years  of 
scientific  study  of  the  relationship  of  test  scores  to  later  school  performance, 
and  a  great  deal  of  statistical  analysis.  Department  of  the  Army  Pamphlet  61.  -2, 
Army  Personnel  Tests  and  Measurement  (111),  describes  these  test  concepts 
in  more  detail. 
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The  tests  that  result  in  the  area  scores  reflect  several  different  kinds 
of  characteristics  of  the  student: 

(1)  Basic  abilities,  identified  through  many  years  of  study 
and  analysis. 

(2)  Previous  learning  of  content  related  to  school  performance. 

(3)  Previous  experience  in  certain  areas,  through  tests  of  knowl¬ 
edge  and  biographical  information. 

(4)  Interest,  as  measured  by  knowledge  and  biographical  information. 
These  tests  are  combined  appropriately  to  maximize  the  prediction  from  apti¬ 
tude  area  to  a  group  of  related  schools  and  training  programs. 

There  is  a  certain  amount  of  correlation  between  the  area  scores.  This 
means  that  students  who  are  high  op  one  area  tend  to  be  high  on  the  others. 
Despite  this,  there  is  more  room  for  differences  to  appear  on  the  several,  area 
scores  than  there  would  be  if  only  one  test  were  used,  as  was  done  during  World 
War  II  with  the  Army  General  Classification  Test. 

The  testing  field  is  one  of  the  oldest  applications  of  scientific  research 
on  people.  Experience  to  date  suggests  that  school  success  can  only  in  part  be 
predicted  from  aptitude  test  scores,  Roughly,  the  accuracy  of  prediction  using 
such  scores  is  not  more  than  twice  that  of  random  prediction,  leaving  at  least 
half  of  the  variation  connected  with  school  success  unpredicted.  This  means 
that  predictions  are  made  in  terms  of  probabilities.  Though  not  certain,  it  is 
more  probable  that  a  high-aptitude  student  will  succeed,  and  that  a  low-aptitude 
one  will  fail. 

Individual  Differences  in  Aptitude 

In  every  course  students  will  differ  in  aptitude,  with  some  able  to  learn 
much  faster  and  more  easily  than  others.  The  extent  of  these  differences  will 
vary  from  one  training  program  to  another.  The  wider  the  spread  of  aptitudes, 
the  greater  the  need  for  adapting  the  training  program  to  students  of  differing 
levels  of  ability.  It  is  important  for  training  officials  to  know  the  range  of 
aptitude  existing  among  their  students, 

A  wide  range  of  aptitude  in  a  group  of  students  creates  severe  prob¬ 
lems  for  a  course  in  which  everyone  proceeds  at  the  same  rate.  If  the  course 
is  pitched  to  the  bright  students,  the  slow  students  will  be  lost;  if  it  is  aimed 
at  the  slow  students,  the  bright  ones  will  be  bored  because  of  lack  of  challenge. 
The  instructional  system  should  be  adjusted  to  take  differences  in  aptitude 
into  account. 

Motivation 

J  „  ' 

Motivation  is  an  important  consideration  in  designing  the  student  manage¬ 
ment  function.  A  student's  motivation  is  believed  by  many  training  officials  to 
be  closely  linked  to  how  much  intrinsic  interest  a  training  activity  has  for  him. 

On  the  other  hand,  many  things  can  be  done  within  the  instructional  system 
to  significantly  increase  the  amount  of  effort  the  students  will  put  forth.  The 
principal  emphasis  of  this  chapter  is  on  those  things  that  generally  can  be  con¬ 
trolled  by  the  system  designer^ 

Fundamental  Ideas  in  Motivation 

(1)  A  motive  is  a  characteristic  that  initiates,  changes,  or  sustains 
behavior.  The  instructional  system  designer  is  concerned  with  initiating  and 
sustaining  effective  learning  behavior  by  the  student. 


P)  M an  iv  a  continuously  wanting  Wing.  If  he  its  satisfied  in  one  area, 
he  develops  needs  in  another. 

(3)  Satisfied  needs  do  not  motivate.  There  naust  he  a  certain  lack  of 
need  satisfaction  to  initiate,  change,  or  sustain  behavior. 

(4)  The  same  need  may  be  satisfied  in  different  ways,,  both. by  the  same 
individual  and  by  different  individual*.  Some  people  who  want;  money  work  for  a 
promotion;  others  rob  banks, 

(5)  A  givert;  mode  of  behavior  may  satisfy  more  than  one  need.  There 
are  several  r  easons  why  an  individual  will  work  hard:  to  learn.  He  may  be  trying 
to  attain  social  prestige  over  other  students*  Or,  he  may  feel'  he  can  gain 

a  promotion. 

*6)  The  individual's  attempts  at  need  satisfaction  will  no*  always,  please 
his  instructor  or  his  supervisor.  Often  an  individual  will  behave  in  veays  that 
may  annoy  or  disturb  his  supervisor. 

Major  Classes  of  Needs 

A  general  framework  in  terms  of  which  the  concept  of  motivation  may 
be  understood  derives  from  Maslow's  (11.2)  characterization  of  the  hierarchical 
nature  of  needs,  and  the  notion  that  it  is  unfulfilled  needs  that  motivate  behavior 
(including  learning  behavior).  Where  high-priority  needs  are  satisfied,  needs 
of  lesser  priority  become  activated,  generally  in  order. 

In  the  first  class,  that  with  the  highest  priority,  are  the  physiological 
needs— for  food,  water,  air,  sleep,  et  cetera.  If  these  needs  are  unsatisfied, 
the  individual  will  be  occupied  with  attempts  to  satisfy  them.  Once  they  are 
satisfied,  the  higher  needs  emerge,  and  will  dominate  the  individual. 

The  next  level  of  needs  is  called  the  safety  needs.  These  include  avoid¬ 
ance  of  physical  harm,  assault,  and  death,  and  a  preference  for  familiar  and 
safe  situations. 

The  next  set  of  needs  is  that  of  the  needs  for  belonging  and  love.  If 
both  physiological  and  safety  needs  are  satisfied,  the  individual  will  strive  for 
social  relationships  and  affection. 

The  next  group  of  needs'  is  concerned  with  self-esteem.  These  include 
desires  for  strength,  mastery,  achievement,  arid  competence.  They  also  include 
respect  and  prestige  given  by  other  people. 

Finally,  there  is  the  need  for  self-actualization,  for  making  the  most  of 
one's  capabilities. 

In  general,  then,  the  needs  most  closely  related  to  intrinsic  motivation 
to  learn  are  relatively  high  in  the  hierarchy.  They  cannot  be  expected  to  appear 
spontaneously  unless  physiological,  safety,  and  belongingness  needs  are  rea¬ 
sonably  satisfied. 

Frequently,  if  an  individual  has  been  deprived  for  a  long  time  of  need 
satisfaction  at  a  certain  level,  these  needs  will  continue  to  dominate  his 
behavior  even  though  the  situation  has  changed. 

Satisfaction  of  the  physiological,  safety,  and  belongingness  needs 
for  a  sufficient  number  of  students  to  enable  them  to  be  motivated  by  higher 
level  needs  may  be  beyond  the  capability  of  the  instructional  system  designer. 

In  a  reasonably  well-ordered  society  the  physiological  and  safety  needs  are 
usually  well  satisfied;  If  they  are,  attention  should  be  directed  to  arranging 
for  the  satisfaction  of  the  belongingness  needs  if  those  concerned  With  self- 
esteem  and  self-actualization  are  to  appear  and  be  the  motivational  basis 
for  training. 
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Previous  Experience  ' 

The  student's  previous  experience  can  significantly  affect  his  motivation 
for  training.  Two  aspects  of  his  experience— education  and  work  experience 
(including  hobbies)— are  especially  important. 

His  particular  educational  experiences  will  have  a  serious  impact  on  the 
student's  attitudes  toward  training  activities  during  his  military  service.  If  a 
student  has  made  good  grades  and  achieved  a  reasonable  degree  of  success  in 
school,  he  will  probably  approach  his  military  training  with  confidence  and  a 
favorable  attitude.  Chi  the  other  hand,  if  he  has  a  long  history  of  academic 
failure,  he  will  probably  approach  his  military  training  with  a  lack  of  confidence 
and  a  severe  distaste  for  studying. 

The  student's  previous  work  experience  and  hobbies,  if  any,  and  the  sports 
in  which  he  engages  can  provide  a  basis  for  making  the  new  material  in  a  course 
more  meaningful.  If  a  relationship  can  be  developed  between  the  new  material 
and  the  student's  previous  experience,  the  new  material  can  be  made  easier  to 
learn.  This  is  illustrated  by  the  fact  that  many  an  instructor  has  found  that 
examples  and  analogies  fell  flat  because  the  student's  background  did  not  permit 
them  to  be  understood. 

General  Implications 

(1)  Each  student  is  unique.  A  training  system  that  receives  a  variety  of 
students,  each  different  from  the  others,  and  exposes  them  to  identical  training 
activities  will  result  in  a  range  of  proficiency  levels.  The  way  to  standardize 
the  results  of  training,  which  is  the  fulfillment  of  the  objective,  is  to  give  each 
student  an  appropriate  amount  of  presentation  and  practice.  Some  will  require 
more  than  others. 

(2)  A  certain  lack  of  need  satisfaction  is  necessary  to  motivate  students. 
Satisfied  needs  do  not  motivate.  It  is  therefore  necessary  to  design  a  situa¬ 
tion  in  which  students  will  have  mildly  unsatisfied  needs,  which  can  be  satisfied 
by  effective  progress  in  the  instructional1  system. 

(3)  Avoid  severe  frustrations  that  activate  lower  level  needs.  If  students 
are  subjected  to  severe  frustrations  of  physiological,  safety,  and  social  needs, 
they  can  be  expected  to  devote  their  time,  energy,  and  attention  to  these,  rather 
than  to  the  content  of  the  training. 

One  kind  of  safety  need  is  the  need  for  assurance  as  to  what  students 
must  learn.  This  can  be  satisfied  by  letting  the  Students  know  the  objectives. 

(4)  Unfavorable  attitudes  toward  training  can  be  changed.  Attitudes  serve 
a  variety  of  functions.  One  function,  as  pointed  out  by  Katz  (113),  is  to  reflect 
the  effects  of  previous  success  and  failure.  People  tend  to  develop  favorable 
attitudes  toward  situations  and  things  that  have  satisfied  their  needs,  and  unfavor¬ 
able  attitudes  toward  those  that  have  frustrated  need  satisfaction. 

Unfavorable  attitudes  toward  learning;  developed  through  failure  in 
previous  educational  experience,  can  be  changed  by  making  the  learning  so  easy 
that  success  is  guaranteed.  This  point  is  supported  by  the  work  of  McKee 
(Ofiesh  and  Meierhenry,  79),  who  has  been  operating  a  self- instructional  school, 
using  programed  instruction,  in  a  correctional  institution.  When  school  dropouts 
began  learning  in  programs  designed  with  a  very  low  error  rate,  a  significant 
increase  in  intellectual  interests  was  noted,  by  both  McKee  and  the  warden.  The 
student  inmates  began  to  ask  for  books  and  use  the  library,  which  few  had  done 
before.  Naturally,  a  reasonable  level  of  aptitude  is  necessary  before  this 
can  happen. 
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REINFORCEMENT 

The  Nature  of  Reinforcement 

Reinforcement  is  the  process  of  providing  reinforcers.  A  reinforcer  is  a 
cue  that  makes  more  likely  the  recurrence  of  the  behavior  a  student  was  engaged 
in  just  before  the  delivery  of  the  reinforcer.  In  other  words.,  the  student  who  is 
doing  something  when  the  reinforcer  appears  is  more  likely  to  do  that  same 
thing  again;  Reinforcers  are  powerful  controls  on  the  behavior  of  individuals. 
Proper  management  of  reinforcement  is  a  very  significant  aspect  of  the  manage¬ 
ment  of  students  function . 

,  '■  It  is  first  necessary  to  decide  what  behavior  is  to  be  reinforced.  This 
behavior  is  the  one  that  leads  to  the  attainment  of  an  objective,  both  task  and 
component  levels,  within  a  reasonable  period  of  time. 

The  second  consideration  is  what  to  use  as  a  reinforeer.  Any  need  satisfier 
can  serve  as  a  reinforeer.  Most  students  in  military  training  situations  find 
they  seldom  have  time  to  do  the  things  they  want  to  do.  Time  off,  then,  is  a 
very  effective  reinforeer.  Rest  periods  are  another  (Azrin,  114).  Such  rein- 
forcers  as  priority  in  the  mess  line,  passes,  and  freedom  from  unpleasant  details 
have  been  fou  id  to  be  effective  (Findlay,  Matyas,  and  Rogge,  115). 

Another  source  of  ideas  for  possible  reinforcers  is  suggested  by  Premack  (1 16), 
Premack  suggests  that  if  Behavior  A  is  more  probable  than  Behavior  B  in  a 
given  situation,  A  can  serve  as  a  reinforeer  for  B.'  Lloyd  Homme  and  his  associ¬ 
ates*  have  taken  this  idea  as  a  guide  for  developing  a  number  of  practical  tech¬ 
niques  for  use  in  reinforcing  progress  in  learning. 

Through  questioning  and  observation  of  students  during  their  free  time,  a 
number  of  highly  probable  behaviors  are  identified  as  reinforcers.  These  behav¬ 
iors  are  listed  on  a  Reinforcing  Event  Menu.  The  Menu  gives  the  student  a 
choice  from  among  several  possibilities.  It  also  avoids  the  possibility  that  one 
particular  activity  selected  by  the  system  designer  may  lose  its  reinforcing 
power  after  continued  use.  ’  „ 

As  applied  by  the  Homme  group,  the  student  successfully  completes  a  cer¬ 
tain  amount  of  learning  activity.  He  then  selects  an  activity  from  the  Menu  that 
he  would  like  to  engage  in,  and  is  permitted  to  do  so  for  a  fixed  period  of  time. 
The  school  is  separated  into  n  reinforcing  event  area  and  a  work  area;  this 
allows  reinforcement  without  disturbing  those  who  are  working.  Students  are 
also  reinforced  for  a  prompt  return  to  the  work  area. 

Another  refinement  is  to  make  a  "contract"  with  the  student.  Each 
reinforcing  activity  has  a  price  placed  on  it,  in  terms  of  the  amount  of  work 
required  before  the  student  can  engage  in  the  activity.  The  student  selects 
the  reinforcing  activity,  and  is  informed  of  the  amount  he  must  learn.  Then, 
when  the  work  is  completed,  he  is  immediately  released  to  engage  in  the 
chosen  activity. 

Despite  the  fact  that  the  student  spends  a  considerable  portion  of  each  day 
in  the  reinforcing  event  area,  the  experience  has  been  that  students  do  much 
more  work  under  these  conditions  than  when  all  of  the  time  is  used  in  work 
on  the  learning  activity.  *  \ 

In  the  sections  dealing  with  the  practice  of  performance  and  practice  of 
knowledge,  the  importance  of  KOR  was  stressed.  In  the  management  of  students 
KOR  sometimes  serves  as  a  reinforeer,  but  sometimes  it  does  not. 
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Reward  and  Punishment  t 

,  .  ’  ” '  f 

Rewards  to  students  for  attaining  objectives  generally  have  a  straightforward  1 

reinforcing  effect.  Punishment,  on  the  other  hand,  has  a  less  predictable  effect. 

Sometimes  it  stops  undesirable  behavior  (but  perhaps  only  for  a  brief  period); 
sometimes  it  actually  reinforces  the  undesirable  behavior.  Often  punishment 
generates  emotional  reactions  that  interfere  with  the  learning  process.  ‘ 

Clearly,  reward  for  attaining  objectives  is  superior  to  punishment  for  not 
attaining  objectives.  Yet,  at  present,  management  of  students  tends  to  operate  ' 

on  the  basis  of  punishment;  when  this  is  the  case,  the  picture  is  one  in  which, 
for  example,  the  student  who  fails  to  work  hard  in  class  or  is  not  attaining  the 
objectives  may  be  restricted  or  placed  on  compulsory  study  hall.  A  more  con¬ 
structive  approach  is  possible  if  the  3tudent  is  able  to  perceive  the  situation  as 
one  in  which  he  will  be  rewarded  for  success,  rather  than  one  in  which  he  will 
be  punished  for  failure. 

Instead  of  using  a  pattern  of  restricting  those  who  do  not  do  good  work,  or 
making  attendance  at  study  hall  compulsory  for  those  who  fail,  for  example,  it 
is  quite  possible  to  operate  in  a  pattern  in  which  ttudy  requirements  are  initially 
placed  higher  for  all  students.  It  is  then  feasible  to  reward  successful  students 
by  giving  them  passes  or  excusing  them  from  portions  of  the  supervised  study, 
while  at  the  same  time  the  less  successful  student  is  required  to  put  in  the 
additional  study  time  without  the  context  of  failing  or  being  punished. 

Schedules  of  Reinforcement 

The  delivery  of  a  reinforcer  upon  the  satisfactory  achievement  of  an  objec¬ 
tive  will  generally  develop  and  sustain  the  behavior  of  working  effectively  toward 
learning.  If  reinforcement  is  delivered  at  each  successful  reponse  by  the 
student,  the  behavior  will  be  developed  quickly. 

If  reinforcement  is  stopped,  then  the  behavior  will  shortly  stop.  This  is 
called  extinction.  Thus  the  problem  for  ,  the  Instructional  system  designer  is 
to  make  effective  learning  behavior  resistant  to  extinction.  The  solution  is 
partial  reinforcement.  At  the  beginning  of  training,  deliver  some  form  of 
reinforcement  very  frequently.  Then  gradually  reduce  the  proportion  of  times 
at  which  the  reinforcement  is  delivered  to  about  one  time  to  each  five  success¬ 
ful  performances.  This  will  sustain  the  behavior  very  well,  and  make  it  very 
resistant  to  extinction,  in  situations  when  delivering  a  reinforcer  every  time 
throughout  training  is  impractical.  One  should  be  prepared  for  some  resent¬ 
ment  on  the  part  of  the  student  when  reinforcement  is  first  skipped. 

The  schedule  of  reinforcement  can  also  affect  the  amount  of  work  performed 
by  the  student.  Ratio  reinforcement  is  given  after  a  certain  number  of  success^ 
ful  performances.  Interval  reinforcement  is  given  after  the  passage  of  a  cer¬ 
tain  amount  of  time.  Ratio  reinforcement  yields  a  high  rate  of  work.  Interval 
reinforcement,  on  the  other  huid,  yields  a  slow  rate  of  work;  not  much  is  done 
early  in  the  period,  but  speed  picks  up  as  time  for  reinforcement  approaches. 

Clearly,  ratio  reinforcement  is  superior  to  interval  reinforcement  for 
management  of  students.  Yet  most  student  "breaks"  in  existing  courses  come 
at  definite  times,  unrelated  to  progress  (for  example,  every  50  minutes). 

ADJUSTMENT  FOR  INDIVIDUAL  DIFFERENCES 

Because  students  are  different,  it  is  impossible  for  all  of  them  to  attain  the 
same  level  of  proficiency  if  identical  learning  experiences  are  provided  for 
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each  student.  For  effective  and  efficient  learning,  it  is  generally  necessary 
to  make  adjustments  for  individual  differences. 

Ability  Grouping 

One  way  of  adjusting  for  individual  differences  is,  of  course,  to  group 
students  by  ability  and  provide  different  courses  for  each  aptitude  level.  The 
feasibility  of  this  procedure  depends  on  being  able  to  identify  students  of  differ¬ 
ing  abilities  in  advance,  and  on  having  enough  students  to  make  multiple 
courses  economical. 

In  the  educational  literature,  ability  grouping  is  referred  to  as  the  track 
system  in  the  United  States,  and  as  "streaming"  in  England.  Results  from 
ability  grouping  have  been  mixed,  and  have  frequently  not  been  effective.  See, 
for  example,  Rudd  (117). 

When  the  performance  of  high-,  low-,  and  medium-aptitude  trainees  was 
compared  after  four  and  eight  weeks  of  basic  training  (Cline,  Reals,  and 
Seicman,  118),  high -aptitude  trainees  in  the  four-week  course  did  as  well  or 
better  than  students  in  the  eight-week  course  in  tests  cf  military  information  and 
performance,  but  not  as  well  in  rifle  marksmanship  and  physical  fitness.  In  this 
case,  additional  training  in  marksmanship  and  physical  fitness  for  the  high- 
cptitude  trainees  might  still  have  resulted  in  a  course  shorter  than  eight  weeks. 

Recycling 

Many  standardized,  fixed-length  courses  really  become  variable-length 
courses  when  students  who  have  failed  a  portion  of  a  course  are  required  to 
repeat  it.  This  repetition  is  called  recycling. 

Recycling,  if  used,  should  be  decided  on  as  quickly  as  possible.  In  courses 
in  which  certain  objectives  are  prerequisites  for  later  work  failure  to  attain 
the  objectives  can  seriously  impede  further  progress.  If  the  decision  to  recycle 
is  made  immediately,  the  work  can  be  repeated  quickly,  and  the  student  may  be 
able  to  progress  without  serious  difficulty.  Delaying  a  decision  may  mean  that 
the  student  accumulates  a  lengthy  failure  experience,  the  effects  of  which  are 
extremely  difficult  to  overcome. 

To  some°exteht  recycling  is  a  punishment,  The  student  has  been  separated 
from  his  friends  and  is  required  to  make  new  ones.  It  is  clear  proof  of  failure, 
and  the  failure  is  obvious  tocthe  new  class,  >j 

Recycling,  of  course,  depends  on  the  ready  availability  of  another  class 
into  which  the  student  can  go.  If  classes  start  only  a  few  times  each  year,  it 
is  not  feasible. 

c  O 

Fixed  Overall  Length,  Variable  Time  * 

Another  way  of  dealing  with  Individual  differences  is  to  establish  a  course 
with  a  given  length,  in  terms  of  days,  but  with  variable  times  spent  in  learning  for 
different  students  each  day.  For  example,  if  the  basic  schedule  required  students 
to  spend  all  day  and  a  few  hours  each  night  in  training,  including  Saturday,  those 
students  who  achieved  the  objectives  in  a  reasonable  time  might  be  excused  from 
study  at  night  and  on  Saturday.  Time  off  can,  thus,  serve  as  a  reinforcer. 

Continuous  Progress 

One  method  of  dealing  with  individual  differences  which  has  been  applied  in 
the  public  schools  is  the  continuous  progress  plan  (Read  and  Crnkovic,  119), 
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sometimes  referred  to  as  the  ungraded  school.  Each  student  learns  at  his  own 
pace.  A  variety  of  techniques  are  used,  each  aimed  at  permitting  the  student  to 
learn  at  the  rate  best  for  his  capability.  In  this  plan,  reinforcers  must  also  be 
provided,  for  speed  of  progress  as  well  as  simply  for  achievement  of  objectives. 

Student- Controlled  Learning 

Several  studies  (Caro,  120;  Milton,  121;  Mager  in  Ofiesh  and  Meierhenry,  79; 
Aldrich,  122)  have  shown  that  mature  learners  do  not  require  formal  classes 
or  managementto  learn  effectively.  College  students  who  were  given  a  semester's 
assignments  to  do  on  their  own  have  done  as  well  as  those  who  attended  class. 
Students  provided  with  specified  objectives  they  must  attain  have  learned  much 
faster  than  those  undergoing  formal  training.  Army  Personnel  Specialists 
learned  as  well  in  this  manner  as  when  attending  classes. 

This  method  requires  certain  arrangements: 

(1)  The  students  should  be  provided  the  training  objectives. 

(2)  All  information  and  practice  materials  required  for  attainment  of 
the  objectives  must  be  made  readily  available  to  the  student  any 
time  he  wants  them. 

(3)  A  rigorous  testing  plan  must  be  devised. 

Student- controlled  learning  might  be  an  especially  feasible  plan  for  officer 
courses.  Officers  are  mature  and  responsible  students.  In  a  given  course,  it 
may  be  likely  that  each  officer  will  have  had  previous  experience  in  a  particular 
area  covered  by  the  course,  but  that  different  officers  will  have  had  different 
experiences.  Thus,  the  amount  of  effort  required  to  meet  a  given  objective 
would  vary  from  officer  to  officer. 

THE  MANAGEMENT  OF  REINFORCEMENT 

Reinforcement  according  to  an  appropriate  schedule  and  tied  to  the  attain¬ 
ment  of  objectives  (within  a  reasonabl  •  time)  can  be  a  valuable  aid  in  motivating 
students.  The  two  principal  problems  associated  with  reinforcement  are  iden¬ 
tifying  appropriate  reinforcers  and  bringing  them  under  the  control  of  the 
instructional  system. 

Identifying  Reinforcers.  Several  possible  reinforcers  have  been  mentioned. 
They,  as  well-as  others,  will  be  listed  here  for  emphasis.  There  are  certainly 
others  that  could  be  identified.  TVro  sources  of  ideas  for  reinforcers  have  been 
suggested— need  satisfiers,  and  highly  probable  behavior.  Specific  reinforcers 
would  include: 

(1)  Rest  periods  and  breaks. 

(2)  Passes.  5 

(3)  Knowledge  of  results. 

(4)  Priority  in  the  mess  hall. 

(5)  Freedom  from  disagreeable  details. 

(6)  Talking  to  other  students. 

(7)  Demonstrating  one's  proficiency  to  others  and  to  oneself. 

Controlling  Reinforcement.  Perhaps  the  main  problem  in  providing  rein¬ 
forcers  in  a  military  training  system  is  bringing  reinforcement  under  the  control 
of  the  training  system.  Frequently  a  separation  exists  between  the  instructional 
system  and  the  agency  responsible  for  the  housing,  feeding,  and  administration 
of  the  students.  Many  of  the  more  significant  reinforcers  are  under  the  control 
of  the  student  company  commander.  To  develop  an  effective  method  for  manage¬ 
ment  of  students  requires  that  these  reinforcers  be  coordinated  with  the  training 


•ystem.  Then  they  will  be  delivered  on  the  basis  of  achievement  of  the  objec¬ 
tives  and  not  on  some  other  basis. 

Unless  this  coordination  takes  place,  it  may  be  possible  unknowingly  to 
punish  success  or  reward  failure.  For  example.  Suppose  that  time  off  is  being 
used  as  a  reward.  The  student  who  receives  the  reward,  and  goes  to  his  com¬ 
pany  area,  may  be  put  to  work;  in  effect,  he  would  be  punished  for  being  success¬ 
ful.  Or  suppose  that  failing  students  are  given  a  considerable  amount  of  free 
time  and  passes  while  awaiting  orders  to  their  next  assignment.  This  would  be 
rewarding  failure,  especially  if  students  who  are  still  in  the  course  are  compar¬ 
atively  restricted. 

Polo  C'deBaca*  has  successfully  used  reinforcement  in  several  institu¬ 
tional  settings  and  mentions  practices  that  militate  against  effective  use  of 
reinforcers.  He  points  out  that  it  is  common  for  the  staffs  unwittingly  to  rein¬ 
force  unresirable  behavior.  They  also  give  reinforcers  away  free,  rather 
than  making  reinforcement  contingent  upon  either  meeting  the  desired  training 
objectives  or  making  effective  progress  toward  those  objectives. 

SEQUENCE  OF  INSTRUCTION 

The  sequence  of  instruction  car,  greatly  affect  the  motivation  of  students. 

It  can  make  the  material  more  meaningful  or  less  so.  Any  added  meaningful¬ 
ness  makes  content  easier  to  learn. 

Functional  Context  Sequencing 

The  preferred  sequence  of  instruction  follows  the  principle  of  Functional 
Context.  This  principle  has  been  tested  in  a  number  of  research  studies,  and 
has  been  shown  to  be  capable  of  reducing  failure,  especially  among  students 
who  are  relatively  low  in  aptitude  (Brown  et  al.„  123;  Goffard  et  al.,  124; 
Hitchcock,  Mager,  and  Whipple,  125;  Shoemaker,  126); 

The  essential  idea  of  Functional  Context  sequencing  is  this: 

(1)  The  student  is  given  a  meaningful  orientation  to  the  entire  job  for 
which  he  is  being  trained.  < 

(2)  Other  topics  are  organized  so  that  the  relevance  of  each  to  the 
whole  job  can  be  demonstrated  to  the  student  at  the  time  it  is  being  taught. 

(3)  A  whole-to-part  sequence  is  followed  in  teaching  the  functions 
of  equipment. 

(4)  Each  student  learns  a  graded,  series  of  job  tasks.  Each  new  task 
to  be  learned  requires  the  student  to  master  new  knowledges  and  skills. 

Thus,  a  context  of  the  job  is  established.  As  the  student  learns  new  tasks, 
the  tasks  are  related  to  the  job  context,  and  the  knowledge  and  skill  components 
are  related  to  the  task  performance  being  learned. 

1  ,o  •  '  *  ■  . 

A  Less  Meaningful  Sequence  ' 

A  less  meaningful  sequence  is  to  begin  the  course  with  a  block  of  "funda¬ 
mentals.**  On  analysis,  these  fundamentals  usually  turn  out  to  be  component 
knowledges  and  skills  to  task  performance  (1).  Task  performance  is  then  taught 
later  in  the  course. 

Some  typical  examples  of  this  practice  are: 

(11  An  introductory  block  on  the  use  of  hand  tools  required  for  later 
task  performance. 

'Presentation  at  the  1966  meeting  o!  the  National  Society  (or  Programmed  instruction. 


(2)  Blocks  of  mathematics,  when  later  certain  tasks  require  computation. 

(3)  Theory  of  the  operation  of  equipment  (e.g.,  electronics  and  hydrau¬ 
lics)  when  later  tasks  call  for  troubleshooting  and  repair. 

Taught  in  this  order,  these  component  knowledges  and  skills  are  taken  out  of 
the  context  of  the  tasks  to  which  they  are  related.  They  are  thus  less  meaning¬ 
ful  and  more  difficult  to  learn.  Students  find  difficulty  in  identifying  the  purpose 
of  this  "fundamentals M  content.  * 

Sometimes  the  fundamentals  c  ontent  is  organized  into  a  subcourse,  which 
feeds  several  courses  devoted  to  instruction  in  skills  required  for  operation  or 
maintenance  of  specific  pieces  of  equipment.  In  addition  to  the  problems  caused 
by  the  lack  of  meaningfulness,  these  core  courses  tend  to  cover  the  maximum 
of  fundamentals  required  by  any  single  equipment-specific  course.  They  thus 
tend  to  be  longer  than  they  need  to  be. 

ASSURING  CLASS  ATTENDANCE 

A  certain  amount  of  excused  absence  from  class  is  unavoidable.  At  the 
same  time,  absences  from  the  learning  situation  that  are  avoidable  do  occur. 
Students  miss  class  because  they  are  on  details,  and  then  have  problems  catch¬ 
ing  up  to  the  rest  of  the  class;  they  may  miss  a  large  part  of  the  day’s  training 
because  they  went  on  sick  call  for  a  minor  ailment. 

Since  a  systems  approach  is  being  taken  in  this  report,  it  should  be  stressed 
that  situations  that  create  unnecessary  absences  should  be  brought  into  coordina¬ 
tion  with  the  training  system.  For  example,  instead  of  students  having  to  go  to 
a  specified  location  for  sick  call,  using  class  time  to  walk  or  wait  for  transpor¬ 
tation,  it  might  be  feasible  to  have  a  mobile  dispensary  visit  the  training  area. 
Personnel  and  other  specialists  might  also  come  to  the  training  area. 

Analysis  of  the  reasons  for  excused  absences  may  suggest  a  variety  of  ways 
to  reduce  impact  of  such  absences  on  the  training  system.  In  one  study  (McCaslin, 
Woodruff,  and  Baker,  86),  scheduling  an  entire  platoon  (or  class)  for  detail 
during  the  same  time  period,  instead  of  operating  a  duty  roster  of  individuals, 
increased  efficiency  of  training. 
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Chapter  6 

AUTOMATION  OF  INSTRUCTION 

AUTOMATION  AND  THE  INSTRUCTIONAL  SYSTEM 
A  Definition  - 

Automation  as  applied  to  instruction  became  a  topic  of  general  discussion 
in  the  middle  1950s,  when  the  teaching  machine  and  programed  instruction  were 
developed.  In  fact,  one  commonly  used  term  for  programed  instruction  is  auto¬ 
mated  instruction.  Naturally,  many  teachers  became  concerned  about  unem¬ 
ployment  resulting  from  technological  advancement  if  automated  instruction 
should  become  widespread.  A  certain  amount  of  confusion  in  terms  has  led  to 
disagreements  and  misunderstandings.  At  the  same  time,  automated  forms  of 
instruction  have  continued  to  develop.  The  intention  in  this  chapter  is  to  clarify 
and  refine  various  meanings  of  the  concept  of  automated  instruction,  and  describe 
some  of  the  forms  it  has  taken. 

Automation  in  an  instructional  system  refers  to  the  performance  of  one  or 
more  functions  of  the  system  by  a  nonhuman  component.  By  this  definition,  there 
is  a  considerable  amount  of  automation  in  the  typical  classroom. 

Two  Extremes 

A  Completely  humanized  (or  ndnautomated)  teaching  system— one  in  which 
all  the  functions  (practice  of  performance,  practice  of  knowledge,  presentation 
of  knowledge,  management  of  students,  and  quality  control)  are  performed  by 
humans— is  extremely  rare.  The  typical  classroom  instructor  uses  many  forms 
of  automation;  training  aids  include  slides,  training  films,  tape  recorders,  and 
television,  Knowledge  or  performance  may  be  practiced  using  workbooks  and 
study  guides.  Tests  are  given  on  a  mass  basis,  and  often  are  scored  by  machine. 
However,  all  of  these  things  are  under  the  control  of  the  instructor. 

In  a  completely  automated  instructional  system  all  system  functions  would 
be  carried  out  by  nonhuman  components,  The  best  example  of  this  is  the  use  of 
programed  instruction  with  no  teacher  present. 

Advantages  and  Disadvantages  of  Automation 

Automation  carries  a  number  of  advantage?.  First,  automated  performance 
of  system  functions  is  invariable*  For  instance,  a  film  shown  over  and  over 
again  will  present  the  same  information  in  the  same  way  each  time.  By  contrast, 
instructors  differ  in  ability,  and  the  same  instructor  will  vary  irom  time  to 
time  as  he  becomes  bored,  tired,  ill,  distracted  by  his  personal  problems,  or 
especially  inspired. 

Second,  completely  automated  instruction  can  provide  effective  training 
under  conditions  where  instructor- centered  classes  are  difficult  to  establish. 

If  the  students  are  dispersed.  If  qualified  instructors  are  in  short  supply,  if 
operational'  commitments  make  it  difficult  to  assemble  classes  at  a  given 
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time— then  automated  "packages”  of  instructional  materials  may  make  it  possible 
to  conduct  effective  training:  despite  these  difficulties. 

Probably  the  major  disadvantage  of  automated  methods  is  that  they  are 
more  difficult  to  revise  to  meet  changing  training  requirements  than  is  the  pres¬ 
entation  by  a  live  instructor. 

Another  problem  is  related  to  the  present  state  of  the  art.  It  is  hot  possible 
to  automate  the  instruction  for  some  objectives,  especially  those  dealing  with 
high  levels  of  skill.  This  may  be  expected  to  change  as  the  results  of  research 
show  that  more  and  more  objectives  may  be  attained  through  automated  means. 

Finally,  automated  methods  may  require  more  preparation  time  and  effort 
than  the  nonautomated  techniques.  However,  if  the  development  costs  can  be 
applied  against  time  and  personnel  savings  when  used  with  large  numbers  of 
students,  automation  can  be  more  economical  than  instructor -based  teaching. 

PROGRAMED  INSTRUCTION  AND  AUTOMATED  INSTRUCTION 


Development  of  Programed  Instruction  Concepts 


Programed  Instruction  (PI)  became  significant  in  the  middle  1950s.  Origi¬ 
nally,  it  referred  to  a  method  of  instruction  in  which  the  teaching  load  was 
carried,  not  by  an  instructor,  but  by  a  specially  prepared  set  of  materials  called 
a  program.  Programs  generally  had  these  characteristics: 

(1)  The  material  to  be  learned  was  carefully  sequenced  and  presented 


in  small  bits, 

(2)  The  student  was  continually  active,  answering  questions  and  so 
solving  problems. 

(3)  The  student  received  immediate  knowledge  as  to  whether  his 
responses  were  cdrrect. 

(4)  The  material  was  made  easy  for  the  student  so  as  to  guarantee  a 
highly  successful  performance  with  few  errors. 

These  characteristics  were  considered  by  many  to  be  the  integral  features 
of  this  type  of  instruction;  however,  research  has  shown  none  of  them  is 
essential.  Successful  programs  have  been  built,  which  violated  each  one  of 
these  presumably  essential  characteristics  of  PI. 

As  a  number  of  aspects  of  the  process  of  program  development  gradually 
became  firm,  a  shift  began  to  take  place,  so  that  PI  was  defined  in  terms  of 
the  process  by  which  programs  were  prepared. 

Program  development  includes  the  following  steps; 

(1)  A  set  of  valid  and  clearly  stated  objectives  is  prepared. 

(2)  A  criterion  test  to  measure  achievement  of  the  objectives 
is  developed. 

(3)  The  content  required  to  attain  the  objectives  is  embedded  in 
appropriate  media,  applying  appropriate  principles  of  learning. 

(4)  The  program  is  tested  and  revised  until  the  students  achieve  the 
objectives  satisfactorily. 

These  steps  describe  in  skeletal  form  the  process  for  the  development 
of  an  instructional  system.  They  do  not  necessarily  imply  automation,  if  we 
assume  that  one  of  the  possible  media  mentioned  in  Step  3  may  be  a  human 
instructor.  Thus,  in  a  very  general  sense,  the  approach  described  in  the  series 
of  reports  by  ihe  present  author  can  be  termed  PI. 

There  is  no  attempt  in  this  report  to  deal  extensively  and  specifically 
with  PI.  A  number  of  references  are  listed  for  the  convenience  of  the 
interested  reader. 
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Automated  Programed  Instruction 

The  practice  of  using  the  terms  programed  instruction  and  automated 
instruction  synonymously  can  be  misleading.  The  practice  evolved  because  the 
earlier  PI  efforts  used  teaching  machines  to  present  the  programs.  The  PI 
programs  may  be  presented  in  textbook  form,  as  well  as  by  teaching  machines. 
However,  human  instructors  may  be  part  of  PI. 

Automated  programed  instruction  refers  to  the  performance  of  instructional 
functions  by  nonhuman  means,  but  within  the  process  of  program  development. 
That  is,  the  media  and  content  must  be  related  to  specific  training  objectives, 
and  should  be  subjected  to  a  criterion,  test  to  make  sure  that  the  materials 
actually  do  teach. 

The  teaching  machine  is  typically  used  to  present  and  practice  knowledge. 
Machines  are  useful  for  a  variety  of  special  applications  that  require  capa¬ 
bilities  beyond  those  of  the  programed  texts.  For  instance,  if  it  is  desired  to 
show  motion  or  sound,  then  a  machine  is  necessary.  Stolurow  (127),  however, 
has  indicated  his  belief  that  the  simple  machine,  which  merely  contains  the 
program,  will  not  hold  its  own  against  the  programed  text. 

Automated  Nonprogramed  Instruction 

The  automation  of  various  instructional  functions  takes  many  forms— training 
films,  texts,  and  workbooks,  for  example.  Historically,  these  items  have  not 
been  programed  in  the  sense  defined  above.  They  have  not  been  developed  to 
accomplish  specific  objectives,  tested  to  determine  whether  they  contribute  to 
the  achievement  of  objectives  ,  or  revised  until  they  do.  Considerable  gain  in 
increasing  the  effectiveness  of  these  "traditional"  training  vehicles  could  be 
accomplished  by  including  them  under  the  concepts  of  programed  instruction, 

THE  COMPUTER  AND  INSTRUCTIONAL  SYSTEMS 

The  Possibility  of  Computer-Based  Instruction 

These  days,  the  ultimate  in  automation  is  the  computer.  Limited  use  has 
been  made  of  computers  in  the  teaching  process,  These  uses  have  been  experi¬ 
mental,  but  more  and  more  can  be  expected  from  this  versatile  device. 

Stolurow  (81)  discusses  the  ■  >le  of  the  computer  in  instruction  and  presents 
ten  critical  requirements  for  a  teaching  machine.  His  discussion  provides  a 
framework  for  considering  the  possible  use  of  a  computer  -based  instructional 
system.  His  critical  requirements  are-summarized  here: 

(1)  Display  (Presentation).  The  material  to  be  learned  is  presented  to 
the  student  by  a  display  unit.  A  variety  of  communication  channels— sight,  sound, 
touch,  or  a  combination  of  these— may  be  used.  In  certain  kinds  of  courses, 
information  may  be  followed  by  a  question  to  answer  or  a  problem  to  solve.  In 
other  situations  complex  skills,  such  as  ilying  an  aircraft,  may  require  that  the 
student  have  faithful  representations  of  significant  environmental  conditions.  The 
display  unit  should  present  both  information  and  knowledge  of  results  to  the  student. 

(2)  Response.  A  component  is  required  to  enter  the  student's  response 
or  action  into  the  teaching  machine.  The  nature  of  the  response  component  will 
depend  on  the  kind  of  response  the  student  is  to  make. 

(3)  Pacing.  The  pacing  unit  is  a  timing  device  that  can  vary  the  time 
intervals  between  the  question  and  the  answer,  or  between  one  question  and 
another.  Or-  it  may  vary  the  speed  of  the  presentation; 
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(4)  Comparator.  The  comparator  analyzes  the  student's  response  by 
comparing  it  with  the  correct  response  stored  in  the  teaching  machine.  The 
pacer  unit  may  affect  the  comparator  when  the  time  of  response  is  a  significant 
aspect  of  the  training  objective. 

(5)  Knowledge  of  Results.  The  correctness  or  incorrectness  of  his 
response  must  be  communicated  to  the  student.  In  some  instances  this  is  a 
simple  matter  of  permitting  the  student  to  compare  his  response  with  the  cor¬ 
rect  one.  In  others,  the  teaching  machine  may  provide  special  signals,  or  addi¬ 
tional  instruction  in  the  case  of  a  wrong  response. 

(6)  Collator- Recorder.  This  component  measures  and  records  the. 
learning  process.  Either  a  collator  or  a  recorder  will  collect  appropriate  data, 
such  as  the  number  of  errors,  types  of  errors,  and  time  required  for  a  response. 
The  recorder  simply  accumulates  data;  the  collator  relates  the  data  to 

the  program. 

(7)  Selector.  The  selector  chooses  the  next  item  in  the  program  for 
presentation  to  the  student.  The  item  selected  may  depend  on  tne  student's 
aptitude  and  previous  learning  record,  thus  permitting  the  teach  .ng  machine  to 
adapt  to  the  student's  requirements. 

(8)  Library.  The  library  unit  stores  information  that,  can  be  released 
to  the  student  as  he  needs  it.  It  may  vary  in  capacity,  access  time,  and  the 
form  of  storage.  The  access  time  of  knowledge  of  results  . is  especially  important. 

(9)  Programing.  A  sequence  of  items  is  required,  whether  predeter¬ 
mined  or  in  a  sequence  that  adjusts  to  the  learner's  progress.  The  programing 
unit  determines  the  sequence  of  learning  activities. 

(10)  Computer.  The  computer  is  almost  a  necessity  if  the  teaching 
machine  system  is  to  be  highly  versatile.  It  can  provide,  depending  on  the 
particular  training  objective,  all  or  many  of  the  foregoing  requirements. 

Thus  the  computer  can  be  viewed  as  a  versatile  tool.  Whether  it  should 
be  used  in  a  particular  setting  should  depend  on  a  cost- effectiveness  analysis. 

An  Example  of  Computer-Based  Instruction 

Most  of  the  experimental  uses  of  computer-based  instruction,  of  which 
there  have  been  a  number,  are  described  by  Coulsen  (128).  The  experimental 
CLASS  system  of  the  System  Development  Ccrporatior  provides  an  example  of 
how  a  computer-based  system  Operates; 

Imagine,  if  you  will,  a  room  with  20  student  stations  and  an  instructor  statior 
The  student  station  consists  of  a  table  on  which  are  a  film  viewer,  tape  recorder, 
and  sometimes  a  book.  There  is  also  a  multiple -choice  response  device,  which 
provides  knowledge  of  results  to  the  student  after  his  response,  in  the  form  of 
signals  that  can  indicate  right,  wrong,  or  the  correct  answer.  With  the  response 
device  the  student  can  request  a  review  or  call  the  teacher  for  assistance. 

The  computer  controls  the  program,  records  student  progress,  and  adjusts 
the  program  to  the  student.  It  evaluates  the  student's  response  and  determines 
what  the  student  should  do  next,  depending  on  the  rules  of  a  given  experiment. 
The  computer  records  the  particular  error  made  by  the  student  and  the  error 
rate.  It  also  records  the  speed  of  response.  Slow  responding  may  indicate 
reading  difficulty,  in  which  case  the  student  may  be  directed  to  an  easier  or 
more  pictorial  program. 

The  teacher  can  monitor  any  student's  work  from  a  control  panel.  If  the 
instructor  wishes  to  help  a  student,  he  can.  The  student  may  call  for  help  from 
the  teacher.  So  may  the  computer,  if  a  student  is  making  too  many  errors  but 
hasn't  called  the  teacher. 
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When  the  instructor  is  called,  he  can  obtain  data  on  the  student's  progress 
that  will  help  him  to  diagnose  the  student's  difficulties.  From  the  teacher's 
station,  the  urious  audio-visual  devices  can  be  controlled. 

In  this  ystem,  a  number  of  the  critical  requirements  described  by 
Stolurow  cai  be  identified.  There  are  displays,  response  devices,  a  comparator, 
knowledge  of  results,  a  collator,  a  selector,  a  library,  a  variety  of  programs, 
all  tied  into,  or  part  of,  a  computer. 

Computer-Based  Information  Systems 

Computers  can  be  used  in  schools  for  purposes  other  than  instruction. 
Various  applications  of  computers  in  the  development  of  school  information 
systems  are  described  by  Bushnell  (129).  Computers  have  also  been  used  for 
scheduling  courses  to  optimize  the  u3e  of  facilities,  for  assigning  students  to 
classes,  and  for  designing  efficient  school  bus  routes. 

’Relevant  to  the  management  of  students  function,  Egbert  and  Cogswell  (130) 
describe  the  design  of  a  Surveillance  and  Detection  System  for  a  school  built 
to  operate  on  a  continuous  progress  plan.  This  system  automatically  monitors 
inputs  describing  student  performance,  detects  trouble,  and  determines  the 
persons  to  be  alerted  in  case  of  trouble. 

Computer  Models  of  Instructional;  Systems 

Another  use  of  computers  is  to  analyze  models  that  simulate  instructional 
systems.  A  model  of  a  school  system  can  be  developed  that  frequently  provides 
a  number  of  advantages  over  observation  and  study  of  the  actual  system,  assum¬ 
ing  reasonable  fidelity  can  be  obtained. 

When  a  system  is  too  large  and  complex  for  observation,  simulation  permits 
study  of  the  system  without  interruption  of  ongoing  activities.  Time  periods 
may  be  expanded  or  compressed.  The  operation  of  the  system  over  a  period  of 
years  can  be  simulated  in  a  few  minutes  in  order  to  study  long-term  trends.  Or 
processes  may  be  slowed  down  for  detailed,  study. 

*  ■  For  training,  simulation  can  show  the  trainee  or  the  decision  maker  the 
advantages  and  disadvantages  of  different  procedures.  The  learner  can  have 
fairly  immediate  knowledge  of  the  results  of  his  actions.  Training  in  the 
environment  of  a  simulated  whole  system  prevents  piecemeal  learning  of  the 
operation  of  the  system. 

If  a  new  system  is  being  designed,  or  changes  are  being  introduced  into  an 
existing  system.  Simulation  can  facilitate  evaluation  of  the  effects  of  the  change. 

The  several  references  by  Bushnell,  Cdgswell,  Egbert,  and  associates 
describe  a  number  of  features  of  computer  simulation  of  school  systems. 
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Part  II 

A  GUIDE  TO  THE  DESIGN  AND 
EVALUATION  OF  INSTRUCTIONAL  SYSTEMS 


Chapter  7 

THE  GENERAL  SYSTEM  PLAN 


It  is  suggested  that  the  initial  stage  in  the  design  of  the  instructional  sys¬ 
tem  be  the  development  of  a  General  System  Plan.  This  plan  should  identify 
the  major  outlines  of  the  system,  including  the  following: 

(1)  Constraints 

(2)  Sequence  of  Objectives 

(3)  Management  of  Students 

(4)  Quality  Control 

CONSTRAINTS 

The  plan  should  identify  the  constraints  that  limit  the  choices  of  the  system 
designer.  The  following  outline  lists  factors  to  be  considered.  Under  each  head¬ 
ing  the  implications  of  each  factor  in  terms  of  constraints  should  be  indicated. 

Constraints 

A.  Overall  Length  of  Course 

B.  Nature  of  Student  Input 

1.  TDY  or  PCS 

2.  Average  aptitude 

3.  Range  of  aptitude 

4.  Previous  experience 

5.  Projected  student  load 

C.  Related  Systems 

1.  Student  support  and  administration 

2.  Supply 

3.  Maintenance 

D.  Cost  Ceilings 

E.  Personnel  Ceilings 

F.  Policies 

G.  Other  Constraints 

THE  SEQUENCE  OF  OBJECTIVES 


The  Functional  Context  Principle 

The  sequence  in  which  objectives  will  be  taught  is  another  aspect  of  the 
General  System  Plan.  The  order  of  the  objectives  should  follow  the  functional 
context  principle  as  discussed  in  Chapter  5.  It  will  be  recalled  that  the  sequence 
starts  with  an  orientation  of  the  student  to  the  job  as  a  whole.  The  relation  of 
each  learning  activity  to  the  total  job  is  emphasized.  The  student  learns  a 


series  of  graded  job  tasks,  each  requiring  him  to  learn  new  knowledges  and 
skills  to  master  the  task.  Finally,  major  pieces  of  equipment  are  taught  in  a 
whole-to-part-sequence,  with  the  function  of  the  whole  equipment  serving  as  a 


53 


context  for  learning  the  function  of  each  major  component,  and  so  on  down  to 
the  parts. 

If  the  system  designer  can  apply  the  functional  context  principle  on  the 
basis  of  this  guidance  and  that  contained  in  the  appropriate  references  to 
Chapter  5,  well  and  good.  If  not,  the  procedure  to  be  described  will  provide  an 
objective  way  of  sequencing  the  objectives  to  achieve  a  close  approximation  to 
functional  context  sequencing. 

A  Technique  for  Functional  Context  Sequencing 

The  first  item  in  the  course  should  be  an  orientation  of  the  student  to  the 
entire  job  for  which  he  is  being  trained.  This  can  be  done  in  .  number  of  ways: 
maintenance  personnel  can  be  taught  how  to  operate  equipment;  work  areas  can 
be  toured,  with  explanations  of  the  duties  to  be  performed,  and  the  major  items 
of  equipment  used;  training  films,  kinescopes,  or  television  can  be  used  to 
present  overview  materials. 

The  remainder  of  the  topics  will  be  placed  in  an  orde  r  based  on  the  fol¬ 
lowing  principles: 

(1)  The  duties  of  the  job  will  be  taught  one  at  a  time,  since  the  tasks 
to  be  performed  under  a  particular  duty  are  related  to  each  other. 

(2)  It  is  important  to  have  the  student  practice  the  performance  of 
tasks  as  early  as  possible. 

(3)  Component  skills  and  knowledges,  once  mastered  in  one  task,  should 
be  used  quickly  in  another  task,  so  that  the  student  can  review  and  practice 
them.  Such  a  procedure  will  favor  overlearning. 

The  Index  of  Task  Complexity 

The  need  for  a  hierarchy,  or  organization  chart,  of  the  component 
knowledges  and  skills  for  each  task  was  pointed  out  in  the  report  on  objectives  (1), 
A  simple  task  has  few  levels  of  components  and  few  components  for  each 
level,  while  a  complex  task  has  many  levels  and  many  components  at  each  level. 

An  Index  of  Complexity  that  will  indicate  the  number  and  level  of  com¬ 
ponent  skills  and  knowledges  can  be  computed  for  each  task  (Figure  2).  Since 
it  is  quite  possible  that  different  analysts  might  define  the  same  job  behavior  as 
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one  broad  task  or  a  series  Of  narrower  ones,  the  index  should  not  be  considered 
as  absolute.  The  technique  is  offered  as  a  suggestion  for  making  a  tentative 
order  out  of  the  total  requirements  of  the  job. 

The  Index  of  Complexity  is  computed  by  multiplying  the  level  (L)  by 
the  number  of  components  (C)  at  that  level,  and  adding  these  products.  In 
Figure  2,  then,  there  are  three  first-level  components;  the  product  of  L  and  C 
is  3.  There  are  six  second-level  components;  the  product  of  L  and  C  is  12. 
There  are  two  third-level  components;  the  product  of  L  and  C  is  6.  The  sum 
of  all  these  products  is  21.  This  task,  therefore,  has  a  complexity  index  of  21. 

A  task  of  low  complexity  has  a  small  number  of  component  skills  and 
knowledges  that  must  be  mastered  before  the  whole  task  performance  can  be 
practiced.  Since  the  student  ought  to  practice  performance  as  early  as  possible, 
it  is  desirable  to  have  tasks  of  low  complexity  taught  first.  However,  tasks  must 
be  considered  in  relation  to  the  duty  of  which  they  are  a  part.  In  order  to  pre¬ 
serve  meaningfulness  in  relation  to  the  job,  all  the  tasks  within  a  single  duty 
should  be  taught  as  a  unit. 

Thus,  speaking  in  a  general  way,  it  can  be  said  that  the  first  duty  to 
be  taught  should  be  that  with  the  lowest  average  Index  of  Complexity  for  its 
composite  tasks;  similarly,  the  first  task  to  be  taught  should  be  the  one  with 
the  lowest  Index  of  Complexity. 

The  Reduced  Index  of  Complexity 

The  first  task  has  been  selected.  How  is  the  next  to  be  selected? 

The  other  tasks  in  that  duty  will  probably  have  some  components  that 
are  identical  to  those  of  the  first  task.  Similarly,  some  tasks  in  other  duties 
may  have  identical  components.  There  is  no  point  in  teaching  these  identical 
components  over  and  over  again.  Therefore,  in  sequencing  teaching,  it  will  be 
necessary  to  identify  these  repeated  components;  and  recompute  the  original 
Index  of  Complexity  (ICo)  for  each  task,  leaving  out  the  repeated  components. 
This  procedure  will  yield  a  Reduced  Index  of  Complexity  (ICr). 

At  this  point,  an  ICr  has  been  obtained  for  every  task  in  the  first  duty 
except  the  first  selected.  It  would  seem  reasonable  to  select  the  task  with  the 
lowest  ICr  to  be  next.  However,  there  is  a  further  matter  to  consider.  In 
selecting  tasks  to  be  taught  next,  it  is  desirable  to  choose  those  containing 
knowledge  or  skill  components  that  have  already  been  taught.  This  will  permit 
further  practice  soon  after  original  teaching,  lessening  the  likelihood  that  the 
student  will  forget  the  componfnt. 

Accordingly,  the  difference  between  the  ICo  and  ICr  is  used  to  take 
this  factor  into  account.  This  difference  is  subtracted  from  the  ICr,  to  yield 
an  Effective  Index  of  Complexity  (ICe). 

To  summarise: 

ICe  *  ICr- (ICo- ICr)  or  ICe-  8-(10-8)  -6 

ICe  =  2 (ICr)  -ICo  Or  ICe  -  2(8)-10  -  6 

The  Effective  Index  of  Complexity  is  used  to  sequentially  select  all  tasks  in  a 
duty  in  order  of  smallest  (least  positive  or  most  negative)  ICe  value. 

A  Summary  of  Functional  Context  Sequencing 

A  rationale  and  an  objective  procedure  for  developing  approximations 
to  functional  context  sequences  have  been  presented.  The  procedural  steps  for 
sequencing  a  whole  job  may  be  summarized  as  follows: 

(1)  Compute  ICo  for  all  tasks. 

(2)  Compute  mean  ICo  for  the  tasks  in  each  duty. 
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(3)  Select  as  first  duty  to  be  taught  the  duty  with  the 
lowest  mean  ICo. 

(4)  Select  as  first  task  in  that  duty  to  be  taught  the  task  with 
the  lowest  ICo. 

(5)  Eliminate  from  all  remaining  tasks  in  that  duty  the  components 
shared  with  the  first  task,  that  is,  compute  ICr  for  the  remain¬ 
ing  tasks. 

(6)  Compute  ICe  =  2(ICr)-ICo  for  all  tasks  except  the  first  in 
the  first  duty. 

(7)  Select  the  task  with  the  smallest  ICe  as  the  next  task  to 
be  taught. 

(8)  Repeat  Steps  (5)  through  (7)  until  all  tasks  of  the  first  duty 
are  arranged  in  sequence. 

(9)  Conapute  mean  ICe  for  remaining  duties. 

(10)  Select  as  the  next  duty  to  be  taught  the  duty  with  the  smallest 
mean  ICe,  and  repeat  Steps  (4)  through  (8). 

(11)  Repeat  the  cycle  until  all  duties  are  sequenced. 

Admittedly,  this  is  a  laborious  procedure  if  performed  by  hand.  With 

proper  coding  and  identification  of  components,  it  could  be  performed  by 
a  computer. 

The  Sequence  of  Objectives  for  the  General  System  Plan 

For  the  purposes  of  the  General  System  Plan,  it  is  sufficient  to  list  the 
orientation  to  the  job,  and  the  tasks  to  be  taught.  The  identification  of  repeated 
components — done  in  the  Functional  Context  sequencing— should  be  retained  for 
later  use  in  analyzing  the  requirements  for  training  equipment,  personnel  and 
material,  and  for  designing  lessons. 

MANAGEMENT  OF  STUDENTS 

The  General  System  Plan  should  identify  the  major  features  of  the  student 
management  function.  (The  material  in  Chapter  5  should  be  used  as  a  guide.) 

The  following  outline  of  topics  may  serve  as  a  suggested  list  of  matters 
to  describe: 

A.  Methods  for  Adjusting  to  Individual  Differences 

B.  Reinforcement 

1.  Types  of  reinforcement  available 

2.  Agency  presently  controlling  each  type 

C.  Controls  Over  Student  Absences 

QUALITY  CONTROL 

The  essential  elements  of  the  quality  control  function  have  been  described 
in  a  previous  report  (2).  These  are: 

(1)  Tests  on  both  task  and  component  objectives. 

(2)  A  method  for  feedback  of  results  of  tests  to  instructional 
system  managers. 

(3)  Procedures  for  taking  appropriate  action  da  the  results. 

(4)  Supervisory  support. 

It  has  been  assumed  that  tests  have  been  prepared,  or  will  be  prepared  early 
in  the  process  of  system  development.  For  the  General  System  Plan,  a  brief 
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description  of  method  by  which  each  of  the  four  essential  elements  of  the  quality 
control  system  will  be  accomplished  will  be  sufficient. 

THE  NATURE  OF  THE  GENERAL  SYSTEM  PLAN 

The  purpose  of  the  General  System  Plan  is  to  provide  an  overall  framework 
for  later  development,  so  that  people  who  are  working  on  specific  aspects  Of 
system  development  will  have  the  same  overall  view  of  the  major  aspects  of 
the  system. 

The  plan  should  be  considered  tentative  and  subject  to  change,  should  the 
later,  more  detailed  work  make  it  advisable. 


Chapter  8 

DESIGN  OF  THE  LESSON 


The  term  lesson  as  used  here  means  all  the  activities  necessary  to 
accomplish  a  given  task-level  training  objective.  The  design  of  the  lesson  is 
described  in  this  chapter.  Topics  discussed  include  identification  of  system 
functions  appropriate  for  specific  objectives,  selection  of  system  components 
to  accomplish  these  functions,  specifications  of  the  lesson,  the  preliminary  test 
of  the  lesson,  and  development  of  the  lesson  plan. 

IDENTIFICATION  OF  SYSTEM  FUNCTIONS 

The  first  step  in  lesson  design  is  to  identify  the  functions  that  must  be 
accomplished  to  attain  the  objective. 

Guidelines  for  Arranging  System  Functions 

Guidelines  for  arranging  system  functions  include  the  following: 

(1)  The  initial  activity  in  teaching  a  job  is  to  orient  the  trainee  to  the 
job.  This  may  be  done  primarily  by  presentation. 

(2)  The  initial  activity  in  teaching  a  new  duty  is  an  orientation  to  the 
duty,  relating  the  duty  to  the  job.  and  describing  briefly  the  tasks  of  that  duty. 
This,  too,  may  be  done  primarily  by  presentation. 

(3)  The  initial  activity  in  teaching  a  task  is  to  provide  a  demonstra¬ 
tion  of  the  performance  of  the  task,  and  to  identify  the  component  knowledges 
and  skills  required. 

(4)  If  a  task  is  of  low  complexity,  with  an  initial  or  reduced  index  of 
7  or  less,  get  the  students  to  practice  immediately  after  the  demonstration. 

(5)  If  a  task  is  Of  greater  complexity,  use  the  functional  context 
method  as  described  in  Chapter  7  as  a  guide  to  sequencing  the  organization  of 
the  objectives. 

(a)  Select  one  of  the  first -level  components  (see  Figure  2). 

(b)  Teach  that  component,  but  starting  from  its  lowest  level, 
and  proceeding  upward. 

(c)  Repeat  for  all  other  first-level  components. 

(6)  As  a  tentative  guideline,  assume  that  once  a  component  has  been 
taught,  it  need  not  be  retaught  to  mastery.  Only  a  reminder  by  presentation  will 
be  necessary.  This  will  not  be  true  in  some  Instances,  but  the  methods  of 
testing  the  lesson  to  be  described  later  in  this  chapter  will  detect  those 
instances  in  which  review  will  be  necessary. 

(7)  Generally,  the  final,  Or  next  to  final,  function  will  be  that  of  quality 
control,  in  which  the  student  demonstrates  his  attainment  of  the  task  objective. 

(8)  In  the  initial  tasks  of  the  total  sequence  the  final  function  in  each 
task  will  be  reinforcement,  for  about  ten  tasks.  Then  reinforcement  should  be 
reduced  gradually  until  a  ratio  of  one  reinforcement  to  every  five  tasks 
reached.  In  assigning  the  reinforcement,  avoid  consistent  patterns  during  any 
ratio.  For  example,  if  the  ratio  is  one  reinforcement  to  every  three  tasks,  do 


not  provide  a  reinforcer  every  third  task,  but  mix  reinforcers  so  that  some¬ 
times  they  are  provided  after  every  fourth  task,  sometimes  after  every  other 
task,  and  so  forth.  A  random  process,  such  as  rolling  dice,  or  a  table  of  ran¬ 
dom  numbers  may  be  used  for  selecting  when  reinforcement  is  given. 

(9)  Immediately  preceding  the  practice  of  task  oi  skill  component 
performance  should  be  the  practice  of  appropriate  knowledge  components. 

(10)  The  orientation  or  "bridge"  content  described  in  the  first  three 
guidelines  need  not  be  practiced. 

Lesson  Function  Analysis 

The  process  of  lesson  function  analysis  is  intended  to  lay  out,  in  appropri¬ 
ate  sequence,  all  the  functions  required  to  teach  a  task. 

As  an  aid  in  performing  this  analysis,  it  is  suggested  that  a  Lesson  Function 
Chart  be  used.  This  is  a  form  that  identifies  the  task  and  all  component  objec¬ 
tives;  it  is  made  up  of  a  series  of  columns,  indicating  phases,  in  which  are 
placed  symbols  representing  the  functions. 

Objectives  which  have  been  taught  previously  are  identified  with  an  asterisk 
to  indicate  they  will  not  be  taught  again,  and  also  as  a  reminder  to  tell  the  stu¬ 
dents  that  this  component  is  required  in  this  task. 

The  lesson  phases  indicate  the  operation  of  the  functions,  in  a  sequence 
rather  than  any  particular  interval  of  time.  Illustrations  of  the  use  of  the  guide¬ 
lines  in  the  preparation  of  a  Lesson  Function  Chart  will  be  presented  in  examples. 

An  Illustration  of  a  Simple  Task 

An  example  of  a  Lesson  Function  Chart  for  a  simple  task  is  shown  in 
Figure  3 .  In  this  task,  the  function  chart  indicates  a  presentation  in  the  first 
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phase,  covering  Lie  task  and  all  its  components.  Then  the  total  task  is  practiced, 
the  student's  proficiency  is  graded,  and  if  successful,  he  receives  a  reinforcer. 

An  Illustration  of  a  Complex  Task 

A  Lesson  Function  Chart  for  a  more  complex  task  is  shown  in  Figure  4.  In 
Phase  1  there  is  a  presentation  symbol,  representing  an  orientation  to  the  task 
and  a  demonstration,  illustrating  Guideline  (3)  In  Phase  2  the  possibility  of  a 
coherent,  reasonably  short  presentation  covering  all  subcomponents  of  Com¬ 
ponent  A  is  recognized.  Phase  3  begins  the  sequence  of  practice  for  Component  A, 
following  Guideline  (5). 

There  is  no  practice  of  knowledge  for  Component  A. 1  .a.;  the  asterisk  indi¬ 
cates  it  has  been  taught  before.  However,  it  has  been  included  in  the  presenta¬ 
tion  of  Phase  2  to  remind  the  students. 

Phases  10,  23,  and  27  are  similar  to  Phase  2,  but  for  other  major  components. 

Phase  26  represents  a  merger  of  presentation  and  practice  of  knowledge  for 
Component  D,  which  is  not  a  complex  major  component. 

Phase  30  represents  the  practice  of  skill  performance  for  Component  E. 

Phases  31,  32,  ar  1  33  represent  practice  of  the  whole  task,  proficiency 
testing,  and  reinforcement,  illustrating  Guidelines  (7),  (8),  and  (9). 

The  points  illustrated  in  the  examples  should  not  be  thought  of  as  rigid 
rules.  The  nature  of  the  organization  of  the  objectives  may  suggest  other  ways 
of  combining  presentation  and  practice  that  will  better  preserve  the  natural 
organization  and  meaningfulness  of  the  objectives. 

The  principal  value  of  the  Lesson  Function  Chart  is  to  lay  out  the  functions 
required  by  a  lesson  in  an  appropriate  sequence  prior  to  specifying  ways  of 
accomplishing  these  functions.  Later  developmental  work  may  make  it  neces¬ 
sary  to  introduce  changes. 

SELECTION  OF  SYSTEM  METHODS  AND  COMPONENTS 

General  Guidelines  for  Accomplishing  Functions 

1  v  . 

Once  the  functions  have  been  identified  in  a  Lesson  Function  Chart,  methods 
and  materials  are  selected  for  the  accomplishment  of  each  function.  Part  I 
has  described  each  of  the  functions  and  indicated  ways  to  accomplish  them  effec¬ 
tively.  There  are  usually  a  variety  of  possibly  effective  methods— some  auto¬ 
mated  and  some  not,  some  euitable  for  large  classes,  some  for  small  classes, 
and  so  forth— depending  on  the  system  constraints. 

The  several  effective  ways  of  accomplishing  a  given  function  will  dif¬ 
fer  in  cost.  In  general,  select  the  least  costly  wajL  Cost  and  effective¬ 
ness  are  discussed  in  Chapter  A,  which  provides  more  detailed  guidelines 
for.  cost  considerations. 

Practice  of  Performance 

Selecting  an  Effective  Method 

The  essential  requirements  for  an  effective  method  for  practicing 
performance  are  an  adequate  simulation  of  the  task  or  skill— based  on  the 
detailed  task  description— and  a  method  for  providing  KOR.  Other  factors 
include  the  practice  schedule,  verbalization,  and  pacing. 

The  principal  possibilities  that  should  be  considered  are: 

(1)  How  should  the  task  be  simulated? 

(a)  With  actual  equipment,  terrain,  etc. 


(b)  With  obsolete  equipment. 

(c)  With  training  devices. 

1^  With  a  whole  task  simulator. 

2  With  a  series  of  part-task  simulators. 

(2)  How  will  the  correctness  of  the  student  response 
be  determined  ? 

(3)  How  will  KOR  be  provided  ? 

(a)  Automatically. 

(b)  By  instructor. 

(c)  By  other  students. 

(4)  Can  practice  be  automated  ? 

The  choices  described  above  should  generate  several  possible  practice 
methods,  which  should  be  described  briefly  and  then  analyzed  for  costs. 

Cost  Considerations 

The  cost  of  each  method  should  be  estimated.  Among  the  kinds  of 
costs  to  be  considered  are: 

(1)  Development  and  procurement,  or  construction 

(2)  Expendable  supplies 

(3)  Replacement  and  maintenance 

(4)  Spare  parts 

(5)  Personnel  costs 

(6)  Storage 

Once  assured  that  the  methods  and  components  are  effective,  determine  the 
cost  per  student  as  a  basis  for  selecting  a  method. 

Practice  of  Knowledge 

Selecting  an  Effective  Method 

The  eaaential  requirements  for  a  method  for  practicing  knowledge  are: 

(1)  A  method  for  presenting  a  symbolic  cue  to  the  student. 

(2)  A  method  for  judging  whether  the  student's  response  is  correct. 

(3)  A  method  for  delivering  KOR  to  the  student. 

The  principal  choices  for  the  practice  of  knowledge  will  include: 

(1)  Self 'Instructional  Methods 

(a)  Programed  learning 

1  Programed  texts 

2  Teaching  machines,  with  appropriate  programs 

(b)  Workbooks,  with  answers  provided 

(c)  Flash  cards,  etc. 

(2)  Classroom  Feedback  Methods 

(3)  Student-Coach  Methods 

Coat  Factors 

Cost  factors  here  are  similar  to  those  for  practice  of  performance: 

(1)  Development  and  procurement 

(2)  Expendable  supplies 

(3)  Replacement  and  maintenance 

(4)  Spare  parts 

(5)  Personnel  costs 
(8)  Storage 
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Again,  the  selection  of  an  effective  method  should  be  based  upon  the  lowest 
cost  per  student. 

Presentation  of  Knowledge 

As  long  as  the  presentation  is  meaningful,  is  properly  spaced  for  effective 
communication,  and  can  be  accurately  received  by  the  student,  there  is  little 
difference  in  effectiveness  between  one  medium  and  the  other. 

Singly  or  in  combination,  the  choices  include: 

(1)  Lectures  or  demonstrations  by  live  instructors 

(2)  Films 

(3)  Television 

(4)  Tape  recordings 

(5)  Books  and  other  written  material 

(6)  Training  ai<js 

Again,  the  selection  should  be  made  on  the  basis  of  the  cost  per  student. 
Proficiency  Measurement 

At  the  conclusion  of  the  practice  of  each  task,  the  student  should  be 
required  to  demonstrate  his  proficiency  in  one  of  two  ways.  The  first  choice 
is  to  observe  the  student  and  score  each  trial  in  the  practice  session.  Then, 
when  the  student  reaches  the  objective,  pass  him.  The  second  choice  is  to  have 
a  separate  proficiency  test.  Procedures  for  developing  accurate  proficiency 
measures  are  described  in  the  quality  control  report  (2). 

The  costs  for  each  choice  depend  on  requirements  for  test  administrators 
and  scorers,  3pecial  equipment  or  test  devices,  and  the  possibility  of  automation. 

Reinforcement 

The  General  System  Plan,  described  in  Chapter  7,  will,  when  developed, 
contain  a  list  of  feasible  reinforcers.  The  choice  of  reinforcers  may  be  based 
on  such  factors  as  cost,  administrative  feasibility,  and  ease  of  application. 

LESSON  SPECIFICATION 

After  the  various  choices  have  been  made  as  to  methods  for  accomplishing 
the  functions,  the  results  should  be  entered  into  a  Lesson  Specification.  This 
is  the  basic  blueprint  for  the  actual  construction  of  the  lesson.  It  will  serve  as 
a  guide  to  requirements  for  personnel,  equipment,  and  supplies.  It  will  indicate 
all  the  devices  needed  for  practice  and  presentation  of  skills  and  knowledge. 

No  particular  format  will  be  prescribed.  However,,  the  following  suggestions 
are  made  as  to  the  content  of  the  Lesson  Specification: 

(1)  Key  the  organization  of  the  specification  to  a  Lesson  Function  Chart. 
Indicate  exactly  how  each  functionfor  each  objective  will  be  carried  out,  by  phases. 

(2)  Include  an  estimate  of  the  number  of  trials,  or  the  amount  of  time 
required  for  each  function  for  each  Objective, 

(3)  Make  a  special  effort  to  identify  exactly  what  the  instructor,  if  any, 
will  be  doing  at  each  phase. 

(4)  For  each  practice  phase,  whether  of  performance  or  knowledge, 
indicate  the  requirement  for  instructions  on  how  to  practice. 

(5)  Indicate  any  references  deemed  necessary  for  use  as  a  guide  to 
good  practice  by  those  who  are  constructing  the  lesson? 

(6)  Indicate  whether  a  particular  item  is  already  available  or  must 
be  developed  or  procured. 


PRELIMINARY  TEST  OF  THE  LESSON 

After  the  lesson  has  been  constructed,  it  is  very  desirable  to  give  it  a 
preliminary  test.  It  should  be  administered  in  accordance  with  the  Lesson 
Specification.  Following  the  lesson,  the  students  should  take  a  test  covering  the 
task-level  and  all  component -level  objectives.  If  a  satisfactory  number  of 
students  meet  the  task-level  objectives,  the  lesson  is  acceptable.  If  not,  exam¬ 
ination  of  the  test  items  dealing  with  component-level  objectives  will  help  pin¬ 
point  specific  problems. 

During  the  preliminary  test,  it  is  also  desirable  to  determine  the  answers 
to  these  questions: 

(1)  Were  there  any  points  at  which  students  were  confused  and  did  not 
know  what  to  do?  This  may  suggest  revision  of  the  instructions,  or  the  addition 
of  new  ones. 

(2)  Were  there  any  periods  during  which  the  instructor,  if  any,  was 
overloaded,  with  so  much  to  observe  or  do  that  he  could  not  perform  his  duties 
effectively?  If  so,  the  load  should  be  lightened. 

(3)  Were  there  any  periods  during  which  the  students'  attention 
wandered?  This  may  suggest  the  need  for  more  attention  to  making  the 
content  interesting. 

(4)  Was  much  student  time  spent  in  getting  ready  to  practice?  This 
may  suggest  the  need  for  preplacement  of  practice  materials. 

(5)  Was  fatigue  evident  at  any  part  of  the  lesson?  This  may  suggest 
the  need  for  a  rest  period. 

If  substantial  numbers  of  students  fail  to  perform  the  task  even  though  they 
pass  the  component  tests,  examination  of  the  need  for  more  task  practice  is 
suggested.  Measuring  proficiency  on  each  trial  will  indicate  whether  more  prac 
tice  than  that  required  by  the  present  lesson  is  necessary. 

If,  in  addition  to  failing  the  task-level  test,  one  or  more  components  are 
also  failed  by  a  substantial  proportion  of  students,  additional  practice  of  the 
components  is  suggested.  Again,  analysis  of  the  amount  learned  after  each  trial 
will  indicate  the  amount  of  component  practice  needed. 

If  the  instructional  system  beipg  developed  is  to  replace  an  existing  course, 
it  may  be  tested  a  lesson  at  a  time  by  obtaining  students  who  meet  the  prerequi¬ 
sites.  On  the  other  hand,  if  the  system  does  not  replace  an  existing  course,  it 
may  be  given  a  preliminary  test  during  the  first  class  of  students  going  through 
the  system. 

THE  LESSON  PLAN 

After  any  revisions  necessitated  by  the  preliminary  test  have  been  made, 
a  lesson  plan  should  be  prepared.  The  lesson  plan  should  include  the  fol¬ 
lowing  information: 

(1)  The  objectives  of  the  lessee  . 

(2)  The  material  and  supplies  needed. 

(3)  A  complete  description  of  all  activities,  in  sequence,  including 
proficiency  measurement  and  any  reinforcers  to  be  provided 
the  student. 
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Chapter  9 

EVALUATION  OF  THE  SYSTEM 


Evaluation  of  the  system  begins  with  the  preliminary  test,  discussed  in  the 
previous  chapter,  and  continues  through  the  life  of  the  system.  Why  the  need 
for  continuous  evaluation? 

First,  the  system  will  change,  as  a  result  of  many  factors.  Personnel  will 
change,  and  as  they  do,  the  new  people  will  change  various  aspects  of  the  system. 
The  changes  may  be  for  the  good,  or  they  may  not.  It  is  important  to  know  the 
effect  of  these  changes. 

Second,  it  may  not  be  possible  to  make  all  the  adjustments  required  by  the 
results  of  the  preliminary  test  before  the  system  is  put  into  operation.  The  sys¬ 
tem  may  require  further  improvement  and  refinement  after  it  is  operational. 

Third,  new  techniques  will  be  developed  that  may  make  the  system  more 
efficient.  These  techniques  will  have  to  be  evaluated. 

Finally,  it  is  practically  impossible  to  keep  the  system  functioning  exactly 
as  it  was  designed.  Changes,  planned  or  unplanned,  will  occur.  The  effective¬ 
ness  of  the  system  can  slowly  diminish  unless  evaluation  is  continuous. 

MEASURING  THE  EFFECTIVENESS  OF  THE  SYSTEM 

Overall  System  Effectiveness 

Techniques  for  measuring  overall  effectiveness  of  the  system  are  included 
in  the  quality  control  function,  and  have  been  described  in  the  quality  control 
report  (2).  Essentially,  the  method  calls  for  development  of  tests  of  the  task- 
level  objectives.  When  these  are  administered  to  the  students,  the  results  indi¬ 
cate  the  effectiveness  of  the  system  in  developing  task  performance.  The 
appropriate  overall  measure  of  system  effectiveness  is  the  proportion  of  objec¬ 
tives  successfully  attained  by  the  students,  or: 

A 

BC 

in  which: 

A  is  the  total  number  of  objectives  attained,  for  all  students 
B  is  the  total  number  of  objectives  measured  by  the  test 
C  is  the  total  number  of  students 

The  Effectiveness  of  the  Lesson 

A  measure  of  effectiveness  of  the  total  lesson  can  be  developed  in  a  manner 
comparable  to  that  of  the  total  system.  Here,  however,  since  mere  is  only  one 
task-level  objective,  the  number  of  objectives  attained  is  divided  by  the  number 
of  students  or: 

A 

C 

in  which  A  and  C  are  defined  as  above. 
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Effectiveness  of  Lesson  Functions 

If  the  lesson's  overall  effectiveness  has  been  shown  to  be  less  than  ib 
desirable,  how  can  the  difficulty  be  pinpointed  ? 

Each  function  of  the  lesson— practice  of  performance  and  presentation  and 
practice  of  knowledge— has  been  designed  to  produce  a  student  who  has  attained 
an  objective,  either  at  the  task  or  the  component  level.  By  testing  students  on 
all  of  the  components  of  the  task,  it  is  possible  to  determine  more  exactly  where 
the  problem  is. 

If  one  or  more  component  objectives  are  not  being  taught,  it  is  appropriate 
to  examine  those  function ~  designed  to  teach  them.  Nearly  every  presentation 
and  each  practice  session  should  bring  some  progress.  The  contribution  of 
each  can  be  determined  by  testing  different  groups  of  students  before  and  after 
each  activity  designed  to  accomplish  that  objective. 

COMMON  SYSTEM  FAULTS 

The  purpose  of  this  section  is  to  identify  a  number  of  reasonably  typical 
system  faults,  as  an  aid  in  correcting  system  deficiencies.  It  may  be  used  as 
a  check  list  by  persons  concerned  with  the  evaluation  and  improvement  of 
instructional  systems.  In  addition,  the  appendix  contains  a  more  comprehensive 
check  list. 

Inconsistency  With  Objectives.  The  various  functions  as  actually  put  into 
practice  may  not  reflect  the  requirements  of  the  objectives.  Students  may  not 
be  practicing  the  appropriate  skills  and  knowledge.  Tests  used  to  measure 
achievement  may  not  be  proper  indicators  of  the  attainment  of  the  objectives. 
Even  the  presentations  designed  solely  to  increase  meaningfulness  may  be 
tested  as  if  they  were  specific  to  course  objectives. 

Inadequate  Consideration  of  Individual  Differences.  Students  of  widely 
varying  aptitudes  and  experience  may  be  required  to  undergo  identical  activity 
during  training. 

Poor  Coordination,  It  may  be  that,  because  of  lack  of  coordination,  all 
aspects  of  the  system  are  not  working  together.  Instructors  may  disagree  with 
their  assigned  role  in  the  system.  Parts  of  courses  may  be  taught  by  different 
departments,  which  may  vary  in  approach  and  effectiveness.  And  the  student 
support  agency  may  differ  with  the  school. 

One  way  of  analysing  the  total  system  is  to  diagram  the  flow  of  infor¬ 
mation  through  it.  The  methods  are  described  in  detail  by  Churchman,  Ackoff, 
and  Arnoff  (131). 

Lesson  Difflciulties.  A  number  of  difficulties  may  occur  within  the  lesson. 
Among  these  are: 

(1)  Lack  of  meaningfulness. 

(2)  Lengthy  sessions  of  presentation  and  student  activity. 

(3)  Little  or  no  knowledge  of  results. 

(4)  Lack  of,  or  ineffective,  reinforcement. 

(5)  Instructor  overload  because  of  too  many  activities  occurring  in  the 
classroom  at  once. 

(6)  Confusion  resulting  from  poor  instructions  to  the  students  as  to 
how  to  practice. 

COST  AND  EFFECTIVENESS 

The  manager  of  the  instructional  system  will  be  concerned  not  only  with 
the  effectiveness  of  the  system,  but  with  its  cost  as  well. 
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The  measure  of  efficiency  suggested  is  that  of  cost  per  objective  attained 
per  student.  This  measure  will  reflect  costs,  the  effectiveness  of  training, 
and  the  number  of  students,  and  will  vary  as  these  factors  vary. 

Examples  of  Cost 

A  matter  of  considerable  importance  to  the  system  manager  should  be  the 
development  of  an  accurate  cost-accounting  system.  Among  the  costs  of  sys¬ 
tem  operation  would  be: 

(1)  Student  salaries,  allowances,  and  other  costs  while  attending  school. 

(2)  Instructor,  supervisor,  and  student  support  salaries. 

(3)  Cost  of  equipment  procurement  and  maintenance. 

(4)  Expendable  supplies. 

In  some  cases,  especially  where  automation  is  being  considered,  student  travel 
would  be  a  relevant  cost.  It  may  be  more  economical  to  ship  an  automated 
training  package  to  the  student  than  to  bring  the  student  to  the  school, 

Among  the  costs  of  development  would  be  salaries  and  allowances  of  all 
personnel  involved  in  planning,  developing,  and  testing  the  system. 

Analyses  of  Cost-Effectiveness  Data 

With  appropriate  cost-effectiveness  data,  a  number  of  possibilities  arise 
for  identifying  ways  of  improving  the  efficiency  of  training. 

Efficiency  may  be  raised  by  lowering  costs  or  by  increasing  the  number 
of  students  attaining  the  objectives.  Separation  of  the  elements  entering  into 
the  cost-effectiveness  ratio  may  suggest  ways  of  increasing  the  effectiveness 
of  training.  If,  however,  the  effectiveness  is  already  high,  special  attention 
should  be  paid  to  costs.  Even  though  the  cost  of  attaining  various  objectives 
will  legitimately  vary  from  objective  to  objective,  comparisons  of  the  costs  of 
various  lessons  will  be  an  appropriate  starting  point  for  explaining  why  varia¬ 
tion  exists.  Perhaps,  some  lessons  need  cost  reduction. 

Analysis  of  different  kinds  of  costs  may  also  reveal  particular  areas  in 
which  cost- reduction  efforts  may  be  especially  fruitful.  In  some  instances, 
for  example,  the  cost  of  equipment  operation  is  a  major  expense.  With  appro¬ 
priate  design  of  training  devices  for  transfer,  it  may  be  possible  to  reduce  the 
amount  of  equipment  operation  required,  and  thus  reduce  costs. 

Cost-effectiveness  analyses,  in  principle  at  least,  show  promise  of  being 
a  significant  factor  in  increasing' the  efficiency  of  Instructional  systems. 
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Appendix 

A  CHECK  LIST  FOR  EVALUATING  TRAINING 

(Correct  Answers  Are  Capitalized) 


.  1.  Obtaining  information  concerning  the  job  for  which  the 
student  is  being  trained. 

a.  Is  there  a  procedure  for  obtaining  information 


about  the  job  ?  .  .. 

YES 

no 

b.  Is  the  procedure  applied  systematically 
and  consistently? 

YES 

no 

c.  Does  the  procedure  collect  performance  informa¬ 
tion  for  meaningful  units  pf  activity  ? 

YES 

no 

d,  Is  performance  information  actively  sought  from 
sources  in  the  work  or  life  performance  situation? 

YES 

no 

e.  Is  performance  information  recorded? 

YES 

no 

f.  Is  performance  information  used  systematically 
and  consistently  to  identify  critical  instructional  needs? 

YES 

no 

g.  Does  the  procedure  provide  complete  coverage 
of  all  likely  aspects  or  occurrences  of  the  desired  work  or 
life  performance  situation? 

YES 

no 

h.  Does  the  procedure  identify  performance  actions, 
condition,  and  standards  relevant  to  the  work  or  life  situation? 

YES 

no 

2.  Identifying  specific  training  objectives. 

a.  Are  decisions  about  what  to  teach  made  on  the 
basis  of  reliable  and  valid  data  ? 

YES 

no 

b.  Are  detailed  analyses  made  of  tasks  to  be  taught 
as  a  basis  for  identifying  knowledges  and  skills  required 
for  task  performance  ? 

YES 

no 

c.  Are  all  skills  and  knowledges  required  for  task 
performance  identified? 

YES 

no 

d.  Do  training  objectives  state  precisely  the  per¬ 
formance  actions,  conditions  and  standards? 

YES 

no 

e.  Do  specific  training  objectives  use  vague  terms, 
such  as  know,  understand,  appreciate,  familiarize,  general 
knowledge,  working  knowledge,  qualified? 

yes 

NO 

3.  Establishing  the  sequence  of  instruction. 

a.  Is  there  an  effective  orientation  of  the  student 
to  the  entire  job  to  be  learned  ? 

YES 

no 

b.  Are  there  blocks  of  skills  and  knowledge  taught 
in  isolation  from  their  use  in  job  tasks  ? 

yes 

NO 

c.  Are  new  skills  and  knowledges  taught  only  when 
required  in  order  to  master  a  new  task  ? 

YES 

no 

d.  Is  the  learning  of  new  knowledge  followed  immedi¬ 
ately  by  practical  exercises  ? 

YES 

no 

e.  Is  the  relation  of  each  new  task  to  be  learned  to 
the  overall  job  clearly  Stated  to  the  student? 

YES 

no 

13 


rrv 


4.  Designing  situations  for  the  practice  of  performance, 
a.  Are  practice  situations  based  on  an  analysis  of 


the  task  to  be  learned  ? 

YES 

no 

b.  Does  the  student  practice  the  entire  task? 

YES 

no 

c.  Has  any  part  of  the  task  been  omitted  from  practice  ? 

yes 

NO 

d.  Do  training  devices  simulate  the  task? 

YES 

no 

e.  Do  instructions  for  effective  use  accompany  the 
training  device  ? 

YES 

no 

f.  Has  the  training  device  been  evaluated  in  terms 
of  developing  student  proficiency  ? 

YES 

no 

g.  Have  training  devices  vs  real  equipment  been 
subjected  to  cost-effectiveness  analysis? 

YES 

no 

h.  Has  the  possibility  of  using  obsolete  equipment 
to  teach  appropriate  skills  been  considered? 

YES 

no 

i.  Do  trainees  receive  frequent  and  immediate  knowl¬ 
edge  of  the  effectiveness  of  their  practice  ? 

YES 

no 

j.  Do  trainees  receive  at  least  one  minute  rest 
between  practice  trials  ? 

YES 

no 

5.  Designing  situations „for  the  practice  of  knowledge, 
a.  Is  the  knowledge  to  be  practiced  clearly  related 
to  an  actual  job  task  ? 

YES 

no 

fc.  Has  information  representing  the  job  cues  pro¬ 
vided  the  student,  and  the  responses  he  is  to  make, 
been  identified? 

YES 

no 

c.  Has  a  practice  session  been  planned  for  ? 

YES 

no 

d.  Have  appropriate  practice  materials  (workbooks, 
self-instructional  programs,  flash  cards,  etc.)  been  designed? 

YES 

no 

e.  Do  trainees  receive  frequent  and  immediate 
knowledge  of  the  effectiveness  of  their  practice? 

YES 

no 

f.  Do  trainees  maintain  a  record  of  their  progress 
during  practice  ?  . 

YES 

no 

6,  Preparing  presentations  to  the  Student. 

a.  Has  the  content  of  the  presentation  been  tested 
on  students  to  determine,  by  means  of  achievement  tests, 
whether  it  communicates  i‘6  the  students  ? 

YES 

no 

b.  Is  the  content  of  the  presentation  meaningful 
to  the  student? 

YES 

no 

c.  Are  there  lengthy  periods  of  presentation  unin¬ 
terrupted  by  practice  ? 

yes 

NO 

d.  Are  films  and  television  Integrated  with 
live  instruction  ? 

YES 

no 

e.  Are  lectures,  demonstrations,  films,  television 
or  tape  recordings  selected  oh  a  cost-effectiveness  basis  ? 

YES 

no 

f.  Have  texts  been  examined  to  be  sure  that  they 
are  within  the  reading  capability  of  the  student  ? 

YES 

no 

7.  Maintaining  student  learning  activity. 

a.  Has  the  degree  of  spread  in  aptitude  scores  of 
the  trainees  been  determined  ? 

YES 

no 

b.  Have  adjustments  been  made  to  the  training 
schedule  to  account  for  differences  in  student  aptitude  ? 

YES 

no 

c.  Have  the  interests>  educational  background,  and 
attitudes  toward  formal  schooling  been  determined  ? 

YES 

no 
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d.  Is  this  information  used  to  make  training  pre- 


sentations  more  meaningful  ? 

e.  Do  students  receive  rewards  that  are  significant 

YES 

no 

to  them  when  they  achieve  course  objectives  ? 

f.  Do  student  rewards  include  those  that  are  under 

YES 

no 

the  control  of  the  student  company  commander  ? 

g.  Has  coordination  been  achieved  with  the  student 
company  commander  to  make  rewards  under  his  control 

YES 

no 

responsive  to  student  performance  in  the  training? 

YES 

no 

fc.  Are  successful  students  punished  ? 

yes 

NO 

i.  Are  failing  or  borderline  students  rewarded  ? 

j.  Has  an  analysis  been  made  of  the  amount  and 

yes 

NO 

reasons  for  excused  absences  from  class  ? 

k.  Have  steps  been  taken  to  reduce  the  amount  of 

YES 

no 

excused  absences  to  a  minimum  ? 

8.  Control  of  the  quality  of  the  training. 

a.  Are  the  tests  direct  translations  of  the 

YES 

no 

training  objectives  ? 

YES 

no 

b.  Is  emphasis  given  to  performance  tests  ? 

YES 

no 

c.  Are  grades  expressed  in  percentage  passing? 

d.  Are  grades  based  on  the  bell-shaped 

YES 

no 

normal  curve? 

yes 

NO 

e.  Are  grades  based  on  percentile  ranks  ? 

f.  Are  test  items  changed  to  make  them  easier  or 

yes 

NO 

harder  to  conform  to  an  "ideal"  distribution  of  grades? 

g.  Are  results  of  student  testing  provided  to  the 

yes 

NO 

instructional  departments  ? 

h.  Do  the  departments  make  changes  in  training 

YES 

no 

procedures  suggested  by  the  results  of  student  testing? 

YES 

no 
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